Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: HALAC (High Availability Lossless Audio Compression) (Read 18954 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #25
Porcus, your statement is interesting to me but I can't believe it until I make my own tests.
I have the reissue of this album by Irond Records. Are we talk about the track:
I Lead You Towards Glorious Times (live)
5:29.800 (14 544 180 samples)

I Lead You Towards Glorious Times yes. I have the Relapse edition. Don't know if Irond is the same, even if number of samples matches (... could be offset). Try the following and see what you get, and see if FLAC's MD5 is D651DE4246A4D5CA64385A91EE9241C5. If not we can exchange files.
flac -2er15 -b16384 --lax
If I afterwards give metaflac --remove-all --dont-use-padding, then I get it down to
51914702 bytes using flac 1.1.4 through 1.4.2
51914703 bytes more using flac 1.4.3.

Here are sizes with other codecs. https://hydrogenaud.io/index.php/topic,122040.msg1010086.html#msg1010086

(Reasons why "MD5 for WAV PCM without metadata" is not reliable, are (1) more than one version of the WAVE file format, and (2) even if you don't see metadata in a tagger, you need to know the size of any JUNK chunk.)

Re: HALAC (High Availability Lossless Audio Compression)

Reply #26
My MD5 is 03813C565FD01FF1D26F8DA56D42A2CB and result is 51 913 118
Writing you a PM...

Re: HALAC (High Availability Lossless Audio Compression)

Reply #27
So, to all who wonder about whether there are several versions of this monster track: these differ in offset only.

Skymmer's has its track boundary starting 664 samples before mine does:

Discarded 664 trailing samples containing non-null values from file #1.
Discarded 664 leading samples containing non-null values from file #2.
No differences in decoded data found within the compared range.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #28
Hello again. I found some time and made some updates.
* Working with small size files
* Working with data containing excessive noise (high entropy)
* Stronger data verification / hashing
* A faster mode

I would like to thank Porcus for making me see some of the points I've missed.
Two small files in Merzbow - Veneorology can now be coded and decoded smoothly. Of course, the compressed size is a bit too much. But in the next stage, we can prevent the increase in size by coding the data as it is for such cases. No problem.
Merzbow - Veneorology
mrz-point1.wav 17,718 bytes
mrz-point1.halac19,823 bytes
mrz-point2.wav35,358 bytes
mrz-point2.halac38.793 bytes

I also prepared a more faster version for speed enthusiasts. Encoder is around 30% and Decoder is around 10% faster.
Sean Paul - Full Frequency - 2014
Wave525,065,800 bytes
Flac-0408,370,552 bytes
Halac-fast396,199,439 bytes
Flac-1395,413,493 bytes
Flac-2394,448,381 bytes
Flac-3391,537,281 bytes

XXXX
Note: I haven't prepared my Linux machine yet.


Re: HALAC (High Availability Lossless Audio Compression)

Reply #30
Tested.
New corpus: track 2 from each of the 38 CDs in my signature, to make it fit a RAM disk, 3h44m of CDDA.  Found out later that the Mahler symphony makes for 1/4 of the total duration.  Should just have deleted that.  Presenting ratios with and without that one, but kept the timings with all.
All FLAC decoding/encoding with 1.4.3 Win32.  With --totally-silent on top everything, it does not spend time spamming the console (which could be significant at these speeds!)
All times are median of at least 3.

6.7 is an impressive figure eh?

w/o M.all 38 tracksencodingdecoding.
60,058%56.30%6.7s 11.6s HALAC FAST
60,063%55.55%8.9s 11.6s FLAC -0b3072 -r0 --no-md5 --totally-silent
59,76%55.36%9.6s 12.5s FLAC -0  --no-md5 --totally-silent
58,10%54.03%9.8s 11.9s FLAC -1b3072 --no-md5 --totally-silent    <---- that is a "smart" joint stereo/dual mono
56,60%52.59%10.0s 13.3s HALAC
56,76%52.49%12.4s 12.7s FLAC -3 --no-md5 --totally-silent <---- that's variable-coefficient LPC but dual mono.
54,951%51.08%14.3s 17.4s TAK at p0 with no WAVE metadata (and no MD5)
54,948%51.05%30 s13.2s FLAC --no-md5 --totally-silent
HALAC isn't so happy about Mahler. Dropping it makes HALAC FAST beat the fastest FLAC at size, and makes HALAC beat FLAC -3 at size. TAK -p0 isn't so happy about that track either, but it is usually so good that it is a mild surprise to see flac -5 beat it at size - they trade about even blows here.

MD5 would add around 4 seconds to FLAC times.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #31
Thank you for the new test, Porcus. I would be glad if I can get information about the operating system and processor you use.
In most cases, Halac-Fast is better than FLAC-0 in the compression ratio. Because I especially set it like that. Of course, there is always space for exceptions.

And once again, these tests are performed using "Ram Disk". In terms of speed, even HALAC(not fast) is always faster than FLAC-0 in normal systems. With ordinary SSDs, HALAC will perform much better in slow and normal systems. My tests aside, I know this from different people who have done numerous tests. In order to address everyone, I do my tests especially with i7 3770k (2012). When I compile with AVX2 supported, I can get a little better results. However, HALAC cannot be used in processors without AVX2 support.

And I added a quick data verification system with V.0.2.0, especially because it was persistently requested. With V.0.2.1, I updated it more powerfully and made it permanent. By performing 2 times more operations. I do not want to disrupt the mechanism again to remove or make it selectable.

During these hashing processes, a very high number of operations take place. When working on the Ram Disk, the cost of these operations will be seen more clearly. However, in slow and normal systems, this transaction cost is not very clear and is not seen. In this test, the other codecs do not make data verification, HALAC is a little behind in this case. So if data verification is not performed, HALAC on Ram Disk will be slightly faster.

Since there is no one else who shares the test results made with a normal system that does not use Ram Disk, I may not be fully understood. There are necessarily testers. I wonder if I'm doing something really bad or harmful, I'm in doubt.

For example, there is a test with an old system:
https://encode.su/threads/4180-hac(high-availabilitation-lossless-audio-compression)?p=81748&viewFull=1#post81748

Re: HALAC (High Availability Lossless Audio Compression)

Reply #32
I'm not sure what you mean here. Rough SSD results will follow, but they are less reliable.

Thank you for the new test, Porcus. I would be glad if I can get information about the operating system and processor you use.
Windows 11, same i5-1135-G7 as in reply 6. Comparison with yours: https://www.cpubenchmark.net/compare/2vs3830/Intel-i7-3770K-vs-Intel-i5-1135G7
It is not in a laptop, it is in a fanless NUC. With 16 GB RAM (as you). When I used RAM disk, I allocated 4 to that.

In most cases, Halac-Fast is better than FLAC-0 in the compression ratio. Because I especially set it like that. Of course, there is always space for exceptions.
Tested the full 38 CDs in my signature, confirms it:
13301342004 bytes for flac -0 -b3072 (encoded with --no-md5 --totally-silent, that's for the timing quoted below), after running metaflac --remove-all --dont-use-padding to get it more comparable ... and to time it.
13224920370 bytes for HALAC_FAST
12739556906 bytes for flac -1 -b3072 --no-md5 --totally-silent
12636918110 bytes for flac -3 --no-md5 --totally-silent
12393304804 bytes for HALAC
12032168564 bytes for flac -5 --no-md5 --totally-silent (-5 is default, and it brute-forces stereo decorrelation)

And once again, these tests are performed using "Ram Disk". In terms of speed, even HALAC(not fast) is always faster than FLAC-0 in normal systems. With ordinary SSDs, HALAC will perform much better in slow and normal systems.
RAM disk seems to give more consistent results. This computer seems to do the following oddity when I loop encoding and then decoding onto the SSD: first run is usually slow, then timing improves, until the CPU starts to throttle.
It isn't the same issue when I test heavier settings. They need to be left overnight anyway, so I can run a couple of warm-ups to stabilize CPU temperature before the actual timing.

You mention you use AVX2. The official flac build does not use AVX2, so I took up the Rarewares 64-bit build that gave best results at https://hydrogenaud.io/index.php/topic,123025.msg1029768.html#msg1029768 for the following.
Encoding using -0fb3072 --no-md5 --totally-silent *.wav (the "f" overwrites, just like HALAC does) and decoding using -df --no-md5 --totally-silent *.flac . This is 38 CDs (one .wav file per CD), on SSD, timed with 7-bench timer64.exe
109 / 93 / 93 / 86 / 86 for five successive FOR-looped encodings. Using "-1" instead of "-0" gives about the same.
159 / 165 / 161 / 171 / 167 for five successive FOR-looped decodings.
Significant variation, I'd say.
But merely writing out the FLAC files takes quite a lot of time too. Running metaflac --remove-all --dont-use-padding takes some 35 to 72 seconds. That just strips out everything except STREAMINFO from the file headers, writes new header and copies the stream into a new file that is written. The big variation there (indicating something going on with allocation and priority in that SSD) is even bigger than in encoding/decoding.

But, now, going to flac -3f --no-md5 --totally-silent *.wav , only three rounds:
100 / 93 / 91 for encoding; 118 / 118 / 109 for decoding.
This would be a giant "WTF!" hadn't it been that it has been observed before that -3 can decode faster than -0. But still it is a surprise.


HALAC then. Since that doesn't support wildcards, and timer64 isn't happy about FOR loops, I put a FOR loop in a .bat file and ran timer64 on that:
153 / 145 / 124 / 115 / 122 for five successive FOR-looped encodings.
297 / 235 / 225 / 221 / 208  for five successive FOR-looped decodings.
I mean, this isn't reliable for anything but establishing that it is slower than FLAC. Encoding speed is around flac.exe -5 --no-md5 --totally-silent. -5 is the default.

Rough HALAC FAST times:
96 then down to 78 for encoding. 155 then down to 134 for decoding.
Not too reliable, but good enough to verify it encodes very fast indeed.



And I added a quick data verification system with V.0.2.0, especially because it was persistently requested.
On the encode stream, so that you can later implement verify without full decode?
Per block? "All" competing formats have something like that, except a couple that suck ... (ALAC has nothing, and TTA claims to offer it, but I know no implementation that actually uses it.)


Re: HALAC (High Availability Lossless Audio Compression)

Reply #33
Thank you for the Linux binaries, particularly the static binary I did need that. This laptop is still on Ubuntu 20.04 so the glibc you compiled with is too old, also the static version should work on non-glibc like alpine linux. Another example that does apply to me is that I rent a Linux server and cannot update the installed libraries, but they do allow running uploaded binaries. glibc is compatible in such a way that if you want to make a highly portable non-static binary you need to use an ancient toolchain, Prime95 for example uses a very old centos to compile (from memory 4?). Personally I wouldn't bother and would just provide the static binary.

Here's some basic testing. Old M.2 SSD, Ubuntu 20.04, laptop with no user input during testing, skylake 6700HQ, fast static Linux binaries.

Code: [Select]
$ time for f in */*.wav;do halac_enc_2.0.1 "$f" "$f.halac";done

real 0m11.848s
user 0m10.116s
sys 0m1.615s

$ time for f in */*.wav;do ~/Downloads/flac143/flac-1.4.3/src/flac/flac --no-md5 --totally-silent -0 "$f";done

real 0m17.009s
user 0m12.440s
sys 0m3.622s


$ time for f in */*.halac;do halac_dec_2.0.1 "$f" "$f.wav";done

real 0m18.359s
user 0m16.082s
sys 0m2.232s

$ time for f in */*.flac;do ~/Downloads/flac143/flac-1.4.3/src/flac/flac --totally-silent -d "$f" -o "$f.fdec";done

real 0m17.172s
user 0m13.083s
sys 0m3.842s

Sizes:
2916907812 wav
2017463945 flac
2016873005 flac stripped of metadata
1994734822 halac
1979508435 sum of best (best of flac/halac unstripped)

I find the large difference in sum of best interesting, both codecs have decent swings in their favour.

edit: Late entry with slac:

Slac to compare to the above:

Code: [Select]
Encode
real 0m27.496s
user 0m24.736s
sys 0m2.417s

Decode
real 0m19.158s
user 0m16.150s
sys 0m2.807s

Size: 1983908820

Re: HALAC (High Availability Lossless Audio Compression)

Reply #34
I have been following this thread with some interest because five years ago I created a codec called SLAC along these lines. It was intended to be as simple as reasonably possible so that it could be easily understood. It was also assumed that it would also be very fast, but it never really lived up to that, at least compared to FLAC.

However, seeing this thread, I decided to take another look and see if I could find something obvious that was slowing it down. I ended up making a couple improvements that give a significant speedup, especially for decoding, which is now significantly faster than FLAC (at least on the system I tried). The compression is approximately equivalent to FLAC -1 when using the -j2 (slowest) encode option.

Experimental branch on GitHub

I don’t know if these codecs are anything more than a technical curiosity, but they may have some use in embedded systems (like QOA). I would like to try creating a lossy version of SLAC that might compete with it.


Re: HALAC (High Availability Lossless Audio Compression)

Reply #35
Porcus, the tests you do are really nice. And I trust your consequences. What I want to say is that HALAC's performance will be better in normal cases, except the use of "Ram Disk". In addition, HALAC also has extra processes for data verification. Flac does not make any verification in the tests. But we're at the beginning of the road and there's things to do.

I do not use AVX2. I think I'm misunderstood. I can only activate the -mavx2 option during the compilation phase. Thus, compilers can automatically make some optimizations. But I do not activate. All current HALAC versions are not compiled as AVX2. So the manual SIMD optimization was not done.
In fact, if I use AVX2, I can switch to rANS. This significantly accelerates the decode time. I do not compile as AVX2 in order to address a wider audience in HALIC.

On the encode stream, so that you can later implement verify without full decode?
Per block? "All" competing formats have something like that, except a couple that suck ... (ALAC has nothing, and TTA claims to offer it, but I know no implementation that actually uses it.)
HALAC makes data verification for each block. Each block is now 6 kb. And yes, if desired, warnings can be given without complete decodes and incompatible blocks can be shown. Of course, this will contain some side information. However, these are subsequent aesthetic add-ons. I have to focus on speed and compression right now.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #36
What I want to say is that HALAC's performance will be better in normal cases, except the use of "Ram Disk".
Why???

RAM disk should have a much higher performance than other forms of storage, if nothing else then simply because it is not accessed over a serial interface.  So why is performance worse?  Is it because you are performing other operations while waiting for disk accessed to complete?
It's your privilege to disagree, but that doesn't make you right and me wrong.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #37
Thank you for the Linux binaries, particularly the static binary I did need that. This laptop is still on Ubuntu 20.04 so the glibc you compiled with is too old, also the static version should work on non-glibc like alpine linux.
I'm running on Linux Mint. However, I could not fully optimize for HALAC in Linux. I just compiled that much.
I also did a test with SLAC and share the screenshots in terms of being convincing. With both i73770k and Ryzen7 5825u. With faster processor and SDD, the SLAC is close to HALAC in Decode time. However, Halac-Fast is faster.

In the meantime, something caught my attention in my different tests. There are problems in some files in SLAC's data verification. So it seems lost. And sometimes in one file, then it can occur in another file. I don't know why the problem is caused. Quite interesting. I added the images in the next post. Doesn't SLAC use data verification?

Seal Paul - Full Frequency - 2014
i7 3770k
HALAC: 383,545,116 bytes (2.985 sec, 3.263 sec)
SLAC: 396,073,220 bytes (6.484 sec, 5.344 sec)
HALAC-fast: 396,199,439 bytes (1.896 sec, 2.844 sec)

Ryzen7 5825u
HALAC: 383,545,116 bytes (1.914 sec, 2.576 sec)
SLAC: 396,073,220 bytes (4.266 sec, 2.591 sec)
HALAC-fast: 396,199,439 bytes (1.306 sec, 2.244 sec)

XXX
XX

XXX

XXX

XX

XX

Re: HALAC (High Availability Lossless Audio Compression)

Reply #38
I have been following this thread with some interest because five years ago I created a codec called SLAC along these lines. It was intended to be as simple as reasonably possible so that it could be easily understood. It was also assumed that it would also be very fast, but it never really lived up to that, at least compared to FLAC.
Hello David.
When I check after the decode process, some files have differences. I came across this situation by chance. I'm adding the images.
XXX

Re: HALAC (High Availability Lossless Audio Compression)

Reply #39
What I want to say is that HALAC's performance will be better in normal cases, except the use of "Ram Disk".
Why???

RAM disk should have a much higher performance than other forms of storage, if nothing else then simply because it is not accessed over a serial interface.  So why is performance worse?  Is it because you are performing other operations while waiting for disk accessed to complete?
I work with minimum things as much as possible for everyone to use the things I have developed. We can say that the biggest bottleneck is hard drives. And all normal users use them. Therefore, I need to particularly take care of the operations on disk access. In other words, it is also very important how a codec uses a fixed disk efficiently. We should not disable this part.

If this constraint is eliminated using RAM disk, Halac does not work badly, but others can close the difference a little. So there is no disadvantage for HALAC. However, since we have to think of normal users in comparisons, I say these in order to achieve more accurate results. Porcus is doing really great tests on his own system and I like it. It has a great accumulation. However, it is nice to see the results from other users.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #40
Yeah, it has been found that file handling had a certain impact on speed.
But IIRC, the differences were not consistently this or that direction ... @cid42 was that you? Corrections?

In addition, HALAC also has extra processes for data verification. Flac does not make any verification in the tests.
That's not correct, FLAC does indeed have two checksums in each frame. They are mandatory in the format. https://xiph.org/flac/format.html#frame_footer , and right above that: there is a CRC-8 for the frame header.

If reference flac encounters mismatch or corruption during decoding, it will throw an error - and then it will exit, unless -F if given. If -F is given, it will keep decoding, throw more errors if any (recent FLAC will mute the frame if it can, older will discard).
Example output from one byte corrupted:
Code: [Select]
1secCORR.flac: *** Got error code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH
*** Got error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC


1secCORR.flac: ERROR while decoding data
               state = FLAC__STREAM_DECODER_ABORTED

The MD5 on the entire stream is atop of that. If no MD5 is present, reference flac will upon decoding give WARNING (which isn't an "error" unless you have used the --warnings-as-errors option).

Re: HALAC (High Availability Lossless Audio Compression)

Reply #41
SLAC might be the fastest decoding of the entire bunch?
130-ish seconds decoding. Encoding takes 170-ish for j0 and j1, 300-ish for j2. Some variability in it though.

Corpus: Same 38 CDs, same SSD. 64-bit encoded j0 slightly faster, so I base on that. I see I should have used -q to get it comparable with HALAC and with flac.exe's --totally-silent, in case console spam is slowing it down.

Putting the sizes into it - and including flac's full stereo decorrelation mode -2 (with -b3072 since I used that for the others)
13301342004   flac -0 -b3072 --no-md5 --totally-silent
13224920370   HALAC_FAST
13133326710   SLAC -j0
12798968972   SLAC -j1
12763832678   SLAC -j2
12739556906   flac -1 -b3072 --no-md5 --totally-silent
12710706259   flac -2 -b3072 --no-md5 --totally-silent
12636918110   flac -3 --no-md5 --totally-silent
12393304804   HALAC
12032168564   flac -5 --no-md5 --totally-silent

Note the benefits from FLAC's stereo decorrelation scheme. At -2 and -5 up, it calculates left, right, mid, side - but it isn't restricted to storing either left+right or mid+side. It can also store left+side or side+right, i.e. "side + whatever is smallest of the three others".
-0 is dual-mono only.
-2 calculates all four for every frame. Saves 590 MB over -2.
-1 has the "-M" switch which does the following: Only every now and then, it calculates all four and selects stereo decorrelation. It keeps this strategy for some frames (calculating only two for those!) and then it does all four again. Gets within 29 MB of -2 that way, i.e. it reaps 95 percent of the benefits for dirt cheap.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #42
I did not know that. Thanks for information.
However, the CRC-8 can only take 256 value. And it is very fast. If we look at the size and value range of the frame, this does not provide a definite security. It is a very superficial measure and collissions are inevitable. I think that's why MD5 is used.
We cannot compare this with the situation in HALAC. I think a much more robust and faster solution for each frame/block is more accurate.
SLAC might be the fastest decoding of the entire bunch?
Does SLAC do data verification? Or did you look at my previous posts?

X

Re: HALAC (High Availability Lossless Audio Compression)

Reply #43
I did not know that. Thanks for information.
However, the CRC-8 can only take 256 value.
CRC-16 in the frame footer. On the entire frame excluding the CRC-16 itself, but including the frame header.

Yes there is also a CRC-8 on the (*counting up*) 64 to 128 bits long frame header. If that is wrong, then the decoder can just ditch the entire frame without even trying to decode it.


As for SLAC I saw that you had posted issues - which are above my skills, I never tried it before yesterday.
Running a rough timing and a few dir or du commands, that is more like me.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #44
flac's frame hashing format is messy, but it does provide "good enough" hashing. It's somewhat more than the 16 bits of crc if you checked fully because there's hell of a lot of structure in there, you mostly know the expected values in the header not that they're needed for the most part anyway, and any corruption in the residual likely results in a different frame size which will cause a desync on the next frame read. Not ideal and not something you'd choose in a greenfield project, but good enough as the status quo. Here's a discussion on flac's strong design decisions to make sync performant, it's related as making sync performant is where a lot of the structure comes from: https://hydrogenaud.io/index.php/topic,123569.0.html

Yeah, it has been found that file handling had a certain impact on speed.
But IIRC, the differences were not consistently this or that direction ... @cid42 was that you? Corrections?
OTOH I had a nightmare of a time getting consistent results benchmarking flac using a nearly-full NTFS HDD on Linux, tl;dr unfixable fragmentation not ideal. It could be mitigated by changing the size of vbuf, basically the fread/fwrite cache size. Be warned meandering thread: https://github.com/xiph/flac/issues/585

Personally I don't really see the benefit of including disk access when it comes to comparing the optimal state of codecs. It's important in the end, but all of these codecs do serial I/O on small chunks of data so they could all be optimised essentially the same way. If there are differences that show up in benchmarking that's just a minor implementation inefficiency that could be fixed down the line, worth noting but IMO not worth showing up in comparison results.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #45
I have been following this thread with some interest because five years ago I created a codec called SLAC along these lines. It was intended to be as simple as reasonably possible so that it could be easily understood. It was also assumed that it would also be very fast, but it never really lived up to that, at least compared to FLAC.
Hello David.
When I check after the decode process, some files have differences. I came across this situation by chance. I'm adding the images.
[attach type=image]28554[/attach][attach type=image]28556[/attach][attach type=image]28558[/attach]
Thanks for testing! I did these speedups using only a couple 2h19m files, and so it's certainly possible that an edge-case bug crept in. I would certainly be surprised if this exists in the master branch. And no, there is no error checking.

Where can I find one of the files that fails, like the Shun Ward track? Thanks!

Re: HALAC (High Availability Lossless Audio Compression)

Reply #46
flac's frame hashing format is messy, but it does provide "good enough" hashing. It's somewhat more than the 16 bits of crc if you checked fully because there's hell of a lot of structure in there, you mostly know the expected values in the header not that they're needed for the most part anyway, and any corruption in the residual likely results in a different frame size which will cause a desync on the next frame read.
Good point. Already by the first 33 bits of a frame, the chance of random bits being valid is like 1:2^19 or something. Of course, that happens like ... hm, once per day of encoded music, as per back of envelope, assuming that everything between frame header and footer "behaves uniformly random".
That is, a decoder that just sits there waiting for a valid frame header will about once per day get something that looks like the beginning of a valid header, but isn't - and then every 256th day get one that will pass the CRC-8.
But if we know it is a real FLAC stream, the decoder will with overwhelming chance get to an actual frame header first, and then it is saved until it drops out. So even if the transmission it is so bad that it drops out every second, then it's gonna take like seven years for it to sync into a false frame first. And if we suppose then that it doesn't check whether it decodes out of the bit depth specified (say with CDDA, the spurious bits don't evaluate to integers outside [-32768, 32767]) - or for some reason doesn't do that (say if it detects a spurious verbatim subframe) then it has to wait all until the frame footer to find out. And if it doesn't buffer until it comes to what it thinks is the footer, you will have 1/10th second of static after having waited for seven years listening to something that skips every second.
Call me when you are done!

Oh wait ... [sic!] ... There are subframe headers too, and then also some bits and combinations are reserved. 5/128 chance of passing through I think? For one subframe. Expected number of channels by random is 42/11 I think?


Not ideal and not something you'd choose in a greenfield project, but good enough as the status quo. Here's a discussion on flac's strong design decisions to make sync performant, it's related as making sync performant is where a lot of the structure comes from: https://hydrogenaud.io/index.php/topic,123569.0.html

Ah, and with a mention of https://wiki.multimedia.cx/index.php/Marian%27s_A-pac
I seemed to recall there was an ancient codec using second-differences only. From 1998. Should be possible to do that fast ... ?


Re: HALAC (High Availability Lossless Audio Compression)

Reply #47
Thanks for testing! I did these speedups using only a couple 2h19m files, and so it's certainly possible that an edge-case bug crept in. I would certainly be surprised if this exists in the master branch. And no, there is no error checking.

Where can I find one of the files that fails, like the Shun Ward track? Thanks!
I uploaded 2 sample wav files. You can try it with these. In addition, it will be better to have a data verification mechanism.

{links removed}


MOD removed links.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #48
Thanks for testing! I did these speedups using only a couple 2h19m files, and so it's certainly possible that an edge-case bug crept in. I would certainly be surprised if this exists in the master branch. And no, there is no error checking.

Where can I find one of the files that fails, like the Shun Ward track? Thanks!
I uploaded 2 sample wav files. You can try it with these. In addition, it will be better to have a data verification mechanism.

{links removed}
Thanks!

I was not able to verify the problem with the first file (12.Legacy), however I was able to verify corruption with the second one (hiphop). The problem only seems to occur on the Windows executables, which makes me think it's something to do with mingw. Anyway, it should not be difficult to track down.

As for the data verification, I agree that having this is absolutely required for a real file format (WavPack has checks on both compressed and uncompressed bitstreams, plus an [optional] whole-file MD5). However that's not what SLAC is. It's just a simple lossless compression library with a demo program. It doesn't even have a file format; the demo program just concatenates the compressed frames (and so there's no seeking or tag support either). Some applications would not require data integrity checks, for example if the frames were sent in network packets that are CRC protected, so this is something that I think belongs in the client using the library to match their needs.

Re: HALAC (High Availability Lossless Audio Compression)

Reply #49
I found the issue with SLAC 0.4 and have uploaded updated executables (now 0.41).

Thanks again for pointing this out! Turns out it was our old friend uninitialized memory, which explains why it worked sometimes. I also fixed a couple things uncovered with UBSAN, but they shouldn't have caused any trouble.