Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Multithreading (Read 23511 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: Multithreading

Reply #125
I don't really know.

It's the first time I deal with dlopen (used in the static OMP library).

At least the executable working properly :)

Re: Multithreading

Reply #126
  • Probably not worth it
  • --mode peakset --blocksize-list 576,1152,1728,2304,2880,3456,4032,4608 --analysis-comp 8p --output-comp 8ep --queue 8192 --tweak 1 --merge 0
@cid42 so I tried really really slow settings for fun on a 6 seconds file (I attached it).

Several times I get an "Error: Init failed" (this error is defined in common.c in flaccid) when using slow settings.
For example if I use this command:
Code: [Select]
flaccid --in 12_-_Napalm_Death_-_You_Suffer.flac --lax --out out.flac --preserve-flac-metadata --queue 8192 --workers 13 --tweak 1 --merge 0 --analysis-apod subdivide_tukey\(3\) --output-apod subdivide_tukey\(6\) --analysis-comp mepl32r15 --output-comp mepl32r15 --mode peakset --blocksize-list 256,512
Ideally I would use more blocksizes in the list (something like `seq -s, 256 256 5120`), but I shortened it to get the error faster (exact same output).
Here is the output:
Code: [Select]
(null)
Processed 1/3
Processed 3/3
Error: Init failed
With that same command, if I limit myself to fewer blocksizes (and bigger step) with `seq -s, 512 512 5120`, it works completely fine and output is created in 155 seconds (not too long if you want to test).

So I guessed flaccid could only work with blocksize >= 512, so I tried different settings (only removed `--queue` arg):
Code: [Select]
flaccid --in 12_-_Napalm_Death_-_You_Suffer.flac --lax --out out.flac --preserve-flac-metadata --workers 13 --tweak 1 --merge 0 --analysis-apod subdivide_tukey\(3\) --output-apod subdivide_tukey\(6\) --analysis-comp mepl32r15 --output-comp mepl32r15 --mode peakset --blocksize-list $(seq -s, 256 256 512)
Still has a blocksize of 256, and "Error: Init failed" but different output:
Code: [Select]
(null)
Processed 1/3
Processed 3/3
tweak(1) saved 20 bytes with 6 tweaks
# ~100 lines like this; then
Error: Init failed
Got this error after 73 seconds. Output FLAC file was almost done: 50ms shorter than input (only one frame is missing ?), MD5 not computed and 2 placeholders in the seektable.


You surely guessed it, I wrote this message in the hope that the error can be fixed.
I also have a question: why is it needed to have all the blocksizes (to bruteforce) multiples of the first one of the list ?
It is surely easier to implement and parallelize, but I find a waste of time when dealing with blocksize steps of 32, 64, 128 (since blocksize from 32 to ~1024 are completely useless to bruteforce).


Many thanks for your software and time.


Re: Multithreading

Reply #127
I'm seeing the same thing here.  I did some limited testing.  In combination with the --lax option,  I can reproduce if the blocksize list has any combination that can equal 256.  If the blocksize list has anything smaller or larger than 256, but doesn't equal 256, there's no error.







Re: Multithreading

Reply #128
You surely guessed it, I wrote this message in the hope that the error can be fixed.
Thank you for making it easy by providing the exact file and settings to reproduce, the latest commit should now work. It looks to be the small (4 sample) partial frame at the end that fails, it looks like it might be failing because the strong lax settings provided can't technically create valid output with such a small frame. I've added a catch-all to the init_static_encoder function that detects when initialisation fails, and tries again with less aggressive settings (presumably ./flac does something similar or the different way it handles the last frame means it doesn't encounter the problem). Not the most elegant solution but it should catch edge cases including this. I can't guarantee that all edge cases are now working, or even that there isn't an underlying problem yet to resolve as this catch-all may just be masking it.

I also have a question: why is it needed to have all the blocksizes (to bruteforce) multiples of the first one of the list ?
It means that regardless of the chosen blocks in a given chain, they always start and end on a discrete boundary that's a multiple of the smallest blocksize. Unconstrained bruteforce is not feasible and an intelligent algorithm that competes well has not been found, peakset is a reasonable compromise.

 

Re: Multithreading

Reply #129
Thanks for the quick update!

It indeed fixes the issue.

I built a static executable once again. Same specs as before: Linux x86-64, generic, static, PGO LTO optimized (O3 flto fprofile-instr-gen), stripped, UPX'd (compressed with -9 --ultra-brute).