Skip to main content

Topic: The (painfully slow) making of foo_dynfil (Read 28638 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.
  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Latest release: foo_dynfil v1 beta 4
This version was released in post #71. Various things were added or changed since the version originally posted below, so check that post and the preceding discussion for the full details.

[original post follows]

Try this and let me know if it works satisfactorily enough: foo_dynra (outdated link removed)

Provides %_dynamic_rating% calculated according to a formula entered under Preferences > Media Library/Dynamic Rating.

While evaluation, an additional field %now% and two functions, $time_diff() for seconds and $date_diff() for days, are supported.
[ETA: Function called with only one argument, like $time_diff(%added%), is a shorthand for difference to %now% (with better performance too).]

Refreshes asynchronously at startup, when you press a button and using bindable main menu command Library > Recalculate dynamic rating.

Prints how much time did a recalculation take to the console, so we can see how silly this idea is.

[ETA: Sample formula to try: http://fb2k.pastebin.com/XrcDzA1j]
  • Last Edit: 30 May, 2013, 10:21:20 AM by db1989
Full-quoting makes you scroll past the same junk over and over.

The (painfully slow) making of foo_dynfil
Reply #1
Some Data:

Library size: 13740 items

%length_seconds% : 0.0426459 s
$muldiv(%play_count%,10000,$date_diff(%added%)) : 0.3555903 s
Above with $if(%play_count%) : 0.1595538 s
FooDAR link : 3.0252005 s
Plays per year (below) : 0.2179175 s

Those numbers are a lot faster than I had expected, although I've only a relativity small library with about 1700 items with a play count. The following code probably isn't very efficient, but the $if(%play_count%) made a big difference (down from about 0.7s.) I wouldn't probably use the fooDAR code but I'm going to see how useful I find Plays per Year.

Good work, the only thing that needs doing is improving "Blah Blah Blah..."

Code: [Select]
$if(%play_count%,
$puts(_dr,$mul($muldiv(%play_count%,10000,$date_diff(%added%)),7,52))
$puts(_s,'0000')
$puts(_sl,$len($get(_s)))
$puts(_m,1$get(_s))
$puts(_l,$len($get(_dr)))
$cut($if($greater($get(_l),$get(_sl)),$substr($get(_dr),0,$sub($get(_l),$get(_sl))).
$substr($get(_dr),$sub($add($get(_l),1),$get(_sl)),$get(_l)),$substr(0.$get(_s),0,
$sub($add($get(_sl),2),$get(_l)))$get(_dr)),$add($get(_sl),2))
)

The (painfully slow) making of foo_dynfil
Reply #2
Refreshes asynchronously at startup, when you press a button and using bindable main menu command Library > Recalculate dynamic rating.


As long as the dynamic rating changes only on a daily basis an option to calculate rating automatically at 0:00 (midnight) would redudantize any need for manual calculation. Even if the rating changes more frequent (since $time_diff retreives seconds) we could have then a sensible rythm of automatic calculations. However, i can't estimate if that is technically feasible.
  • Last Edit: 24 September, 2010, 02:19:46 PM by q-stankovic
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #3
Wow! Most unexpected and thanks a lot Yirkha.

Firstly, I suggest we stop referring to foo_DAR and call it DAR, short for DADA (Date and Duration adjusted) Auto-Rating (for the moment at least).

Secondly, I'm going to play around with this when I get a chance to. In the meantime I've posted below the latest DAR Code (which is much improved and solves pretty much all the issues that were raised earlier in this thread).

Because anyone messing with this is a sophisticated fb2k user, I'll let you change whatever needs to be changed to work with the latest playback stats and Yirkha's foo_dynra (the code below uses foo_customdb):
Also the © is actual a c enclosed in brackets --- ( c ) but the forum software changes it to ©.   
Code: [Select]
$puts(c,%play_counter%)
$puts(x,$add($cwb_datediff(%added%,%cwb_systemdatetime%),2))
$puts(y,$cwb_datediff(%added%,%last_played%))
$puts(z,$sub($get(x),$get(y),2))
$puts(l,%length_seconds%)
$puts(d1,$muldiv($add($get(l),540),1,4))
$puts(d2,$muldiv($get(l),$get(l),9000))
$puts(d3,$add($get(d1),$get(d2)))
$puts(r0,$mul($add(1000,$muldiv($get(d3),$get©,100)),10))
$puts(dd,$div($add($get(y),50),10))
$puts(pp,$muldiv($get©,10000,$get(x)))
$puts(2,$muldiv($get(dd),$get(pp),100))
$puts(3,$muldiv($get(x),135,100))
$puts(4,$div($get(pp),50))
$puts(5,$div($muldiv($add($cwb_datediff(%added%,%first_played%),2000),500,$add($div($get(d3),6),70)),$add($mul($get©,$get©),1)))
$puts(6,$muldiv($get©,625,$get(x)))
$puts(7,$add($get(3),$get(5),$get(6)))
$puts(r1,$add($get(2),$get(r0)))
$puts(r2,$add($get(4),$sub($get(r1),$get(7))))
$puts(r3,$sub($get(r2),$div($mul($get(r2),$get(l),$get(z),$get©),500000000)))
$puts(r4,$ifgreater($get(r3),0,$get(r3),1))
$ifgreater($get©,0,$num($get(r4),5),-----)

Thanks again Yirkha, will report back when I can.

C.
  • Last Edit: 25 September, 2010, 12:18:31 AM by carpman
PC = TAK + LossyWAV  ::  Portable = Lame MP3

  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Reply #4
Good work, the only thing that needs doing is improving "Blah Blah Blah..."
OK, thanks.
Naturally the dialog text will provide some more useful information in the final release.
I had the component sitting abandoned on my harddisk for more than a year and was reminded of it only now when carpman linked to this topic in another thread. So this is just a quick beta release, published only because I've been in a publicization mood recently.

As long as the dynamic rating changes only on a daily basis an option to calculate rating automatically at 0:00 (midnight) would redudantize any need for manual calculation. Even if the rating changes more frequent (since $time_diff retreives seconds) we could have then a sensible rythm of automatic calculations. However, i can't estimate if that is technically feasible.
Anything is possible.™
I'm just not sure about the midnight part - somebody might get used to it being synchronized at 0:00 and then bitch after refreshing manually or at startup, suddenly with a non-aligned timestamp.
But yes, some automatic refresh while fb2k is running is definitely needed - then the component will "just work" for most people and that's always a good thing.
I guess I'll simply add two options, "
  • Recalculate at startup" and "
  • Recalculate each [day____]" (with a list of various periods).

    Wow! Most unexpected and thanks a lot Yirkha.
    You are welcome. (And please don't kill me for the delay described above...)
    I suggest we stop referring to foo_DAR and call it DAR, short for DADA (Date and Duration adjusted) Auto-Rating (for the moment at least).
    OK, well it's your formula. My part is only providing a generic framework to accomplish things like this - so the component will probably get its own thread later too. Maybe you could create a new, up-to-date topic for DADAAR too after everything settles down.
Full-quoting makes you scroll past the same junk over and over.

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #5
Maybe you could create a new, up-to-date topic for DADAAR too after everything settles down.

Yes, precisely - I will do. I'll come up with something that isn't so fooDADADADAAAAAAAAAAAR. 

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3

  • NEMO7538
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #6
Yirkha, a couple of technical questions just out of curiosity :
  • where is %_dynamic_rating% stored ? I guess not in tags
  • how did you (which SDK function) manage to make this info available to the rest of foobar (playlist renderer, search engine, ...). I thought this capability was removed some time ago

Thanks

  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Reply #7
1. where is %_dynamic_rating% stored ? I guess not in tags
It's naturally not stored anywhere, it's calculated. And cached in memory till the next recalculation.

2. how did you (which SDK function) manage to make this info available to the rest of foobar (playlist renderer, search engine, ...).
metadb_display_field_provider. Nothing new, all playback statistics, custom DB, foo_exvar and other components have been using it for a long time.
Full-quoting makes you scroll past the same junk over and over.

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #8
Okay, well this DAR Code works with the official Playback Statistics component and Yirka's foo_dynra:

Code: [Select]
$puts(pc,%play_count%)
$puts(x,$add($date_diff(%added%),2))
$puts(y,$date_diff(%added%,%last_played%))
$puts(z,$sub($get(x),$get(y),2))
$puts(l,%length_seconds%)
$puts(d1,$muldiv($add($get(l),540),1,4))
$puts(d2,$muldiv($get(l),$get(l),9000))
$puts(d3,$add($get(d1),$get(d2)))
$puts(r0,$mul($add(1000,$muldiv($get(d3),$get(pc),100)),10))
$puts(dd,$div($add($get(y),50),10))
$puts(pp,$muldiv($get(pc),10000,$get(x)))
$puts(2,$muldiv($get(dd),$get(pp),100))
$puts(3,$muldiv($get(x),135,100))
$puts(4,$div($get(pp),50))
$puts(5,$div($muldiv($add($date_diff(%added%,%first_played%),2000),500,$add($div($get(d3),6),70)),$add($mul($get(pc),$get(pc)),1)))
$puts(6,$muldiv($get(pc),625,$get(x)))
$puts(7,$add($get(3),$get(5),$get(6)))
$puts(r1,$add($get(2),$get(r0)))
$puts(r2,$add($get(4),$sub($get(r1),$get(7))))
$puts(r3,$sub($get(r2),$div($mul($get(r2),$get(l),$get(z),$get(pc)),500000000)))
$puts(r4,$ifgreater($get(r3),0,$get(r3),1))
$ifgreater($get(pc),0,$num($get(r4),5),-----)

By the way, Yirkha, am I right in thinking that your component doesn't allow fields such as $date_diff(%added%) outside of the component, i.e. you can't use fields such as %now% as a stand-alone bit of script in a column?

[EDIT]
I've got some code (which basically uses the DAR as an index, for those who prefer dots instead of numbers). I'll post that up when I start a new thread.

C.
  • Last Edit: 25 September, 2010, 03:01:17 PM by carpman
PC = TAK + LossyWAV  ::  Portable = Lame MP3

The (painfully slow) making of foo_dynfil
Reply #9
But yes, some automatic refresh while fb2k is running is definitely needed - then the component will "just work" for most people and that's always a good thing.

While experimenting with carpmans script i noticed that another type of automatic updating would be senseful. When a currently playing track increases its playcount and therefore changes its first played time then its and only its dynamic rating could be updated to reflect immediately the change of the value.
  • Last Edit: 25 September, 2010, 03:40:36 PM by q-stankovic
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #10
I agree, that would be ideal.
By the way Yikha, great job!

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3

The (painfully slow) making of foo_dynfil
Reply #11
It is possible now to have a field for autorating or hotness or average daily playcount or ... But you have to decide for one of them. 
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #12
Yes, so what you're suggesting to Yirkha is something like having, the same interface but with 3 or so tabs to enter code, and having 3 or so fields for outputting:

%_dynamic_rating1%
%_dynamic_rating2%
%_dynamic_rating3%

Then of course you could see side by side how the different ratings formulas stack up against oneanother.
DAR and Hotness (AFAIK) are doing two different things, DAR is an attempt to level the playing field and give you an idea of your favourite tracks (new and old, long and short, right now) and Hotness is telling you your latest craze (I think).

C.

EDIT: Personally, after setting up fb2k 1.1 with the official Playback Stats and Yirkha's dynra component, what I'd like is:
1) The ability to see simple things like Age (in no. of days since added) and (no. days since last played) (i.e. $date_diff(%added%)) -- that's very handy for testing and a 2 or 3 digit number takes up much less space than a long date and time stamp/string.
2) I've had to install foo_customdb.dll (and remove all fields except Playcount) just for the "count as played after x%" (but that's an official playback stats issue).
3) The ability to transfer stats from unofficial stats components (foo_custominfo.dll) to the official one. It would be nice if someone could work on an SQL export to XML (?) or whatever the official format is for importing stats. EDIT2 -- I've got stats for DAR testing that are more than 2 years old - thus I'm still with 0.9.4.3 when it comes to checking how the DAR formula performs. Ultimately you want everyone using the official stats plugin (I'd like to personally - and in my test setup I'm using most of it - but that count as played after 1 min bugs me).
  • Last Edit: 26 September, 2010, 12:06:43 PM by carpman
PC = TAK + LossyWAV  ::  Portable = Lame MP3

The (painfully slow) making of foo_dynfil
Reply #13
%_dynamic_rating1%
%_dynamic_rating2%
%_dynamic_rating3%


Hmm, that would be better:

%_hotness%
%_auto_rating%
%_daily _playcount%
%_days_since_added%

Own names!
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #14
Yeah, but each time someone comes up with a new thing Yirkha would have to add a field for it - not so good. This way there's no worry of redundancy.

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3

The (painfully slow) making of foo_dynfil
Reply #15
I thought of creating own name s for field as you can do for columns in CUI - have you ever waited for musicmusic ...
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #16
have you ever waited for musicmusic ...

What do you mean?

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3

  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Reply #17
Am I right in thinking that your component doesn't allow fields such as $date_diff(%added%) outside of the component, i.e. you can't use fields such as %now% as a stand-alone bit of script in a column?
Yes, those are supported in the local context only. It hasn't been possible to add custom global $functions() since [a href='index.php?showtopic=62286']0.9.5.2[/a]. As for the %now% field, the local availability is obviously the only way to avoid the well known problems (repaint glitches and other unreliability).


While experimenting with carpmans script i noticed that another type of automatic updating would be senseful. When a currently playing track increases its playcount and therefore changes its first played time then its and only its dynamic rating could be updated to reflect immediately the change of the value.
Indeed, seems reasonable. I'll add that to the component, so that when any metadata changes, the dynamic values of the relevant track(s) will be also recalculated.

And I'll make it do the same for any newly added tracks too. Currently the only action taken was to remove information for tracks removed from the Media Library.

I had been pondering for a while whether the data should be updated using %now% of the last recalculation or the new, current time in such cases - because people might not want some random play count increase or whatever change the time reference point. But then I strongly decided for the latter, because 1) the recalculation time is not predictable anyway, 2) nobody can expect people's script working properly with negative date/time differences, 3) it will be the right thing to do to return correct values in almost all usages anyway.


It is possible now to have a field for autorating or hotness or average daily playcount or ... But you have to decide for one of them. 
Yeah, yeah. Of course I knew that someone will suggest that, it's not in the current release only because it needs additional UI, restarting fb2k upon change and so on. (If anybody is wondering about why a restart is necessary, it's a price we pay for the performance to not [a href='index.php?act=findpost&pid=564566']suck[/a] anymore.)

So, for a likely tiny decrease in performance, support for multiple fields will come in the next version too. And naturally this way:
%_hotness%
%_auto_rating%
%_daily _playcount%
%_days_since_added%
Own names!


But another question is then raised by extending the functionality - can it still be called foo_dynra for Dynamic Rating if people use it to show things like "Added: 123 days ago" in their playlists?
I say no, therefore the component will be called "Dynamic Fields" from now on. I guess.


Quote from: tedgo link=msg=0 date=
In contrast with the new Playback Statistics 3.0, foo_dynra observes only tracks from the Media Library.
Yes, and it will always stay like that. For the best performance during drawing, sorting and so on, the component needs to remember the calculated value for all tracks it will be used with. Limiting that to the Media Library, which is always fully loaded and clearly constrained seemed like a decent solution to me.


I'd appreciate it, if a moderator would close this thread.
If you don't mind, I'd leave it alive for a few more days to continue our discussion about foo_dynra, as I don't like complicated topic splitting. After release of the component, I'd create a proper stand-alone topic for it and finally close this.
Full-quoting makes you scroll past the same junk over and over.

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #18
Yirkha, no problem, of course I don't mind. Keep it open for as long as is needed.
By the way, I'm very pro all the decisions for foo_dynra. Very good news. 
Now if Peter allows count as played only after x% in his foo_playcount, I do believe a choir of angels will descend and lift me to the nirvana of my choosing.

Cheers,

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3

  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Reply #19
I wouldn't count on that, though. I'm pretty sure it uses the playback_statistics_collector API, which has a well defined constant behaviour - "at least 60s of the track has been played, or the track has reached its end after at least 1/3 of it has been played through". The core does the monitoring and calls registered components when the condition is hit. Extending this so that each client could change the parameters would need a new version of the API as well as making the inner implementation more complex. And Peter is not usually keen on messing with stuff which works well enough.
Full-quoting makes you scroll past the same junk over and over.

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #20
"at least 60s of the track has been played, or the track has reached its end after at least 1/3 of it has been played through"

I see. So Peter could actually make it even more simple, by simply fixing it at 50% or 67%. So a) still, no one gets to "change the parameters", and b) that's more simple than the current "60 secs or ...".

And it would get rid of the strange part where 39 mins of a 40 min track (97.5%) are skipped, yet it's still counted as played. 

Anyway, this is OT for this old thread.

Thanks again, and I'm looking forward to the next episode of Dynamic Fields starring Yirkha.

C.

EDIT: Spelling.
  • Last Edit: 29 September, 2010, 11:40:19 PM by carpman
PC = TAK + LossyWAV  ::  Portable = Lame MP3

The (painfully slow) making of foo_dynfil
Reply #21
@carpman
A total play length for a track would solve that. You already explained somewhere that total play length diveded by track length then would be the "real" playcount. The relation between that playcount and the official one would also offer some information about skipping behaviour. I liked the idea when i read it the first time.

@yirkha
Is that possible in the context of your component at all? It seems to me that it would work a bit different than the dynamic fields: %total_play_length% has to be available in scripts which define a dynamic field.
  • Last Edit: 30 September, 2010, 02:51:42 PM by q-stankovic
german support forum: www.foobar-users.de / user: qwert73

  • Yirkha
  • [*][*][*][*][*]
  • Moderator
The (painfully slow) making of foo_dynfil
Reply #22
Recording how many seconds a particular track has been playing in total needs persistent storage, track pinning scheme and all the other stuff Playback Statistics do. I'm not going to create yet another foo_playcount_mod_mod from this component.
Full-quoting makes you scroll past the same junk over and over.

The (painfully slow) making of foo_dynfil
Reply #23
I see! I anyway had the feeling it would be an other thing than a dynamic field - just: such feelings doesn't help an average user to recognize which features are feasible and which not. 
german support forum: www.foobar-users.de / user: qwert73

  • carpman
  • [*][*][*][*][*]
  • Developer
The (painfully slow) making of foo_dynfil
Reply #24
Recording how many seconds a particular track has been playing in total needs persistent storage, track pinning scheme and all the other stuff Playback Statistics do. I'm not going to create yet another foo_playcount_mod_mod from this component.

Yes, indeed, it's ideal in some ways, but not really, precisely because it's also a big deal (as Yirkha points out) and also it becomes unnecessary when playcount is counted after a fixed % (e.g. 67%).

If you eat a few peas you can't say you've had dinner, half the meal? - maybe, but two thirds? - I think you can politely excuse yourself as being satisfied; you've made a decent effort.

romor's case is an ideal example, he chose the mother of all feasts (155 min song), took a bite and foo_playcount excused him from the table. 

C.
PC = TAK + LossyWAV  ::  Portable = Lame MP3