Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: FLAC v1.4.1 (Release) (Read 18983 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Re: FLAC v1.4.1 (Release)

Reply #50
I tried using --totally-silent on Linux, got similar results as when not using that option.

Re: FLAC v1.4.1 (Release)

Reply #51
Tested again and found something even more shocking.
* Output to terminal window takes a hell of a lot of time - and differs over -p or not.
That explains a lot. When you add -p, the progress bar is updated more often.

What happens if you test in another shell, for example Powershell?
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.1 (Release)

Reply #52
Got pretty much the same as with cmd. Tested: PowerShell 5 and PowerShell 7, command
Get-Date ; C:\bin\flac-1.4.1-win\Win64\flac.exe -2pf *.flac ; Get-Date
and then without the p and then adding --totally-silent to either.

There was one thing I forgot to wrote, which might be consistent with this progress bar story: the first "New computer tested" item in reply 47 was a 96 kHz signal, so it probably spends then more time encoding the signal and thus the percentwise penalty looks smaller than the two next. In reply 49 I also used a 96 kHz signal, but then -0 and -0p, so actual encoding takes less. Makes sense?


And there we have it? An encoder that spends as much time displaying progress as it does on the actual job - hey in the age of New Public Management that sounds like just ideal  ;D

Seriously this needs a fix: users who select the fastest options wouldn't want to spend thirty percent more time just watching numbers fly in the Windows shell. (Thirty percent penalty is without the -p. Carelessly putting a -p into there, that was was when you could get a doubling.)


(... and my issues understanding what -p is supposed to do and when, are at somewhere else ... oh eff, you have probably explained stupid detail to me before, so here goes. I thought it worked as follows:
After the "guesstimation", the windowing function is selected; using whatever method, a "raw floating-point predictor" vector is generated and subsequently rounded to 15 bit precision because well that is a legal value. For precision 15, do the Rice encoding, and then successively round to precisions of 14, 13, ..., 5 and do the Rice encoding; one of these will encode to the smallest of size - and that is selected. Did I get it reasonably right?
Oh BTW a minor point to the documentation and to flac --explain: they miss the information that -q is restricted to <=15; but, who uses -q manually anyway?)

 

Re: FLAC v1.4.1 (Release)

Reply #53
Seriously this needs a fix: users who select the fastest options wouldn't want to spend thirty percent more time just watching numbers fly in the Windows shell.
But the problem is in Windows shell. not in the way FLAC does input/output. The problem isn't present on non-Windows, and I just tested in MSYS (3rd party shell in Windows), the problem is much, much less pronounced there: around 3% instead of 100% for preset -0p.
Music: sounds arranged such that they construct feelings.


Re: FLAC v1.4.1 (Release)

Reply #55
But it does run! The performance problem is in how that terminal renders text apparently. What do you propose: FLAC packing it's own terminal window because Microsoft can't be bothered to fix problems?
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.1 (Release)

Reply #56
I just cloned the latest commit, but it gives me the 10th as the date.
flac git-797c9c5e 20221010

Nevermind, I was working from 2 terminal windows, and forgot to refresh one before installing.

Re: FLAC v1.4.1 (Release)

Reply #57
In Windows, writing to the terminal window is a synchronous event and the writing application cannot continue execution until that event is completed, hence the delay. The only way to avoid the delay is to minimise, or eliminate, the writes! That's been an issue since forever.


Re: FLAC v1.4.1 (Release)

Reply #59
In Windows, writing to the terminal window is a synchronous event and the writing application cannot continue execution until that event is completed, hence the delay. The only way to avoid the delay is to minimise, or eliminate, the writes! That's been an issue since forever.
That doesn't explain why MSYS is 30 times as fast as Windows' native shell.
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.1 (Release)

Reply #60
I'm guessing, but may be the MSYS shell buffers the screen IO?


Re: FLAC v1.4.1 (Release)

Reply #62
What do you propose: FLAC packing it's own terminal window because Microsoft can't be bothered to fix problems?
Limit the progress output to say, two updates per second. Who needs more anyway?
Feels bad to have to workaround issues especially unnecessary ones that wouldn't exist if MS were better, but this is probably the best solution. Alternatively tell users that really care to use the Linux binary on windows through WSL2, that may bypass the issue. Either way cathartic alcohol and cursing should be involved, effigy burning ritual optional.

Or maybe powershell doesn't have the problem? If the problem is only in the legacy cli, which MS likely wants to let die, such an amusing issue existing at all starts to make more sense.

Re: FLAC v1.4.1 (Release)

Reply #63
Alternatively tell users that really care to use the Linux binary on windows through WSL2, that may bypass the issue.

If we are at the "tell user" stage of solutions, it is much easier to tell them use the --silent switch (-s). They will get a "progress update" when each file is done. Heck, that could be in the help file. Or --fast could be changed to be synonymous to -0s  :P

But seriously:
From reply 51, FLAC does control how often updates are printed - different ways by different parameters. It just does it in a way that is bad when compression is fast, then it adds an overhead of thirty percent.
Since that is something FLAC has made an active choice to do, it might as well do it a bit less counter-productive?

The same thing seems to happen with -e - and certainly, with -pe. Enjoy this sadistic one for twentyfour-fold time consumption:
Code: [Select]
C:\tmp>c:\bin\timer64 c:\bin\flac-1.4.1-win\Win64\flac.exe testflacfile.flac -0pefs -b 17

Commit   =     14364 KB  =     15 MB
Work Set =     14924 KB  =     15 MB

Kernel Time  =     1.218 =   22%
User Time    =     4.187 =   77%
Process Time =     5.406 =   99%
Global Time  =     5.419 =  100%

C:\tmp>c:\bin\timer64 c:\bin\flac-1.4.1-win\Win64\flac.exe testflacfile.flac -0pef -b 17

flac 1.4.1
Copyright (C) 2000-2009  Josh Coalson, 2011-2022  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

testflacfile.flac: wrote 113408171 bytes, ratio=1,000

Commit   =     14360 KB  =     15 MB
Work Set =     14928 KB  =     15 MB

Kernel Time  =    30.859 =   23%
User Time    =     7.468 =    5%
Process Time =    38.328 =   29%
Global Time  =   129.796 =  100%

To compare, -8pe took only 77 seconds.


(Huh, what does -0e do? Exhaust the Rice4 vs Rice5 options?)

Re: FLAC v1.4.1 (Release)

Reply #64
PS H:\> measure-command {h:\flac -0f image.flac}
TotalSeconds      : 38.4290222

PS H:\> measure-command {h:\flac -0fs image.flac}
TotalSeconds      : 31.8444154

PS H:\> measure-command {h:\flac -0fpe image.flac}
TotalSeconds      : 81.9119224

PS H:\> measure-command {h:\flac -0fpes image.flac}
TotalSeconds      : 34.5344766

PS H:\> measure-command {h:\flac -t image.flac}
TotalSeconds      : 22.332029

PS H:\> measure-command {h:\flac -ts  image.flac}
TotalSeconds      : 19.4594858

====================================================

PS H:\> measure-command {h:\wavpack -f image.wav}
TotalSeconds      : 32.6553532

PS H:\> measure-command {h:\wavpack -fq image.wav}
TotalSeconds      : 32.6964023

PS H:\> measure-command {h:\wvunpack -v image.wv}
TotalSeconds      : 53.021008

PS H:\> measure-command {h:\wvunpack -vq image.wv}
TotalSeconds      : 52.9964522

Re: FLAC v1.4.1 (Release)

Reply #65
Writing output incurs significant speed penalties at -5 and -7 as well - depending on computer and on track lengths, it seems.


* 18 percent speed penalty at default -5 in this first test:
Same Acer laptop as I have used before on the file set in my signature, .wav to .flac. Looped through the 38 CD corpus in my signature multiple times with -5f and then with -5fs.
-5: 407 seconds plus/minus three
-5s: 345 seconds plus/minus two, and fan much louder.
(Then a -5 again, fan spins down and 412 seconds.)

The seconds penalty is about the same here though:

* As previous, but with -7.
-7: 580 seconds-ish
-7s: 515 seconds-ish.

* Now on to the Dell laptop, I did something different: re-encoded (flac -> flac) a bunch of single track rather than one per CD.
-5: 375 seconds-ish
-5s: 345 seconds-ish
The Dell varies in time, but the impact stays around 30 seconds on average and in "neighbouring runs".

* Final test I did was then -7 on the Dell, and although I did measure differences, the fan soon started playing tricks on me. Something between 470 and 500 and something between 12 and 35 seconds differences.
It would have looked nice if I got the same difference there too, but ... too unreliable.

Re: FLAC v1.4.1 (Release)

Reply #66
(MSYS is not available as portable application it seems.)

Well, you don't need MSYS, you only need mintty, the actual terminal emulator.

However, mintty itself is only installable as part of either Cygwin, MSYS2, or Git for Windows. I suggest the third option, because it has by far the lowest footprint, on account of not being a full development environment unlike the first two options.

In either case, this is not a FLAC problem and therefore not something FLAC needs to fix.

Re: FLAC v1.4.1 (Release)

Reply #67
In either case, this is not a FLAC problem and therefore not something FLAC needs to fix.

Uh. If you are releasing an .exe for Microsoft Windows, and it turns out the verbosity level you chose is so outrageous that it bogs down the application this much - who else do you think should fix that?

As just pointed out by @bennetng, the competition handles this just fine. (I have written it before, the more testing I do the more I like WavPack.)

Re: FLAC v1.4.1 (Release)

Reply #68
Without investigating how the codes are actually written, I can see that the cursor in flac jumps around. I guess it either constantly set the cursor position or rewrites every character including the non-numerical ones which don't need to be updated.

Re: FLAC v1.4.1 (Release)

Reply #69
It writes things like "97% complete, ratio=1,000", and it writes a whole damn lot of it.
Attached what you get by throwing 2>suitableoutputfilename.txt to the end of it. For comparison, two WavPacks at different settings -fmv --blocksize=128 and -hx.

Anyone who really needs this much spam?

Re: FLAC v1.4.1 (Release)

Reply #70
As has been noted already, the Windows command-line terminal is the only terminal which doesn't handle FLAC's verbosity properly, which means that FLAC's verbosity level is quite sane. As such, this:

the verbosity level you chose is so outrageous that it bogs down the application this much

is a bit inadequately worded.

First of all, as just explained, the verbosity level is not "outrageous".

Second, it is not FLAC's verbosity level that bogs down the application, but the application's incompetence at handling pretty much the only thing it needs to be able to handle as a terminal.

If you are releasing an .exe for Microsoft Windows, you expect that power users who are going to be running things in a terminal themselves will know how to work around Microsoft's incompetence. That, after all, is what one signs up for when deciding to be a power user on Microsoft Windows.

In other words, just use mintty. 🤷

Re: FLAC v1.4.1 (Release)

Reply #71
(...) the verbosity level you chose is so outrageous (...)
(...) and it writes a whole damn lot of it.
(...)
Anyone who really needs this much spam?
My problem here is that you seem to imply FLAC devs are stupid because they didn't notice the Windows terminal takes as much time writing 25 characters as FLAC takes processing 80.000 samples.

FLAC has had two main developers vanishing without saying good bye, and the last of those two didn't really have much love for the project. This means *many* things in flac are as they were left 20 years ago. I *try* to get things updated, but this is one of those things that was at one point tuned to CPUs as they were back then. I'm trying to get a new release out soon because it turns out I broke gstreamer when I implemented better error handling (to mimick other lossless codecs) and I won't get a fix for this in in time, because a fix where stats are output based on time instead of samples processed needs a lot of testing.

So please be gentle. I try to keep up. I just get really frustrated when something stupid like this, which I feel isn't FLACs fault, is treated as such anyway because 'everyone uses Windows, you should adapt to its stupidity'
Music: sounds arranged such that they construct feelings.

Re: FLAC v1.4.1 (Release)

Reply #72
I guess there's no problem with hardware accelerated Windows Terminal?
I also guess that it shouldn't be a problem how much you output to terminal, but how often. Once, several times a second should be enough for most people...

Re: FLAC v1.4.1 (Release)

Reply #73
It writes things like "97% complete, ratio=1,000", and it writes a whole damn lot of it.
Attached what you get by throwing 2>suitableoutputfilename.txt to the end of it. For comparison, two WavPacks at different settings -fmv --blocksize=128 and -hx.

Anyone who really needs this much spam?
PS H:\> measure-command {h:\flac -0fpb128 image.wav  -o 128.flac} | select totalseconds
image.wav: wrote 424069445 bytes, ratio=0.547

TotalSeconds
------------
  33.1922087


PS H:\> measure-command {h:\flac -0b4608 image.wav  -o 4608.flac} | select totalseconds
image.wav: wrote 395690573 bytes, ratio=0.510

TotalSeconds
------------
   3.7869877


PS H:\> measure-command {h:\flac -t 128.flac} | select totalseconds
128.flac: ok

TotalSeconds
------------
   6.6325306


PS H:\> measure-command {h:\flac -t 4608.flac} | select totalseconds
4608.flac: ok

TotalSeconds
------------
   2.6697964

I used Notepad++ to open the logged text files and found something like "79% complete, ratio=0.990" followed by 25 backspaces, and this pattern is repeated 119 times in the file. Cutting the repetition should speed things up a lot.

Also, -t does not have "ratio".

X

Re: FLAC v1.4.1 (Release)

Reply #74
My problem here is that you seem to imply FLAC devs are stupid because they didn't notice the Windows terminal takes as much time writing 25 characters as FLAC takes processing 80.000 samples.
My problem here is the attitude that the problem is irrelevant because it only occurs in the Windows shell - as if anyone could ever expect Windows CLI application to be run in a Windows shell, those who run command-line on MS-Windows must be expected to use something else even without being told.

A simple "well sucks, file an issue at github and then I estimate this to enter the to-do list at position 666 - meanwhile use -s" would at least acknowledge that this is an actual issue, and would most certainly get you more "fair enough!" responses back - at least one more, I can personally vouch for that.

Also a "quickest fixes not likely to be rushed, this will be fixed if and when someone can do it properly" would signal that hey, we are not back in 2012 when everyone was worried that the new maintainer would scrap Windows support altogether. Which would be a problem, take it from one who has spent years in emacs/tcsh/bash (in that order).