Skip to main content
Topic: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript) (Read 5709 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Note: Do NOT uninstall foo_playcount! This component does not replicate all that component's functionality, and actually performs better if foo_playcount is also installed.

I wrote a little component to record the timestamp of every play of every song (after the component was installed). This component provides three fields right now, all of which return JSON formatted arrays.

v2.0.0 released and Last.fm support is here!

New fields provided:
  • %played_times% - Date formatted list: ["2012-08-04 15:58:37", "2012-12-10 14:40:46", "2018-01-02 23:38:13"]
  • %played_times_js% - JS timestamp list: [1344117517000, 1355172047000, 1514957893431]
  • %played_times_raw% - raw foobar timestamps: [129885911170000000, 129996456470000000, 131594314934316754]
  • %lastfm_played_times% - Date formatted list of scrobbles: ["2012-08-04 15:58:37", "2012-12-10 14:40:46", "2018-01-02 23:38:13"]
  • %lastfm_played_times_js% - JS timestamp list: [1344117517000, 1355172047000, 1514957893000]
  • %lastfm_play_count% - Count of last.fm plays, a la %play_count%: 5
  • %lastfm_added% - Single date: "2012-08-04 15:58:37"
  • %lastfm_first_played% - Always exactly the same as %lastfm_added%. Use whichever one makes most sense logically
  • %lastfm_last_played% - Single date: "2018-04-04 15:58:37"

To retrieve last.fm scrobbles correctly, head to the component properties page and enter your last.fm username. No authentication is required. Last.fm interaction is fairly complex, there is detailed help on the configuration page and more descriptions in this thread if you need them.

Unfortunately, there's no way to write a component that honors the $meta_num/$meta(prop, index) standard (because it's not a meta field) so to maximize use of this component you will really need to have a JScript/WSH panel where you can actually loop through the JSON arrays and do fancy things. IMO, %played_times_js% and %lastfm_played_times_is by far the most useful because you can actually do stuff with it, i.e. after retrieving the value, looping through and calling new Date() on it. Will add some example code for that.

Note: If you have foo_playcount installed already, then the first time the song is played after adding this component, it will also append the %first_played% and %last_played% values to the array. Sorry if you've already played a song 68 other times, there's no way of knowing when those times were. Edit: If you were scrobbling those songs, now there is!

Note: If you have a version installed prior to v.1.0.2, you will have to manually delete it from your components folder! The filename has changed from foo_enhanced_playback_statistics to foo_enhanced_playcount. Foobar will fail to launch if both are installed at the same time. Please delete the entire foo_enhanced_playback_statistics folder.

Latest version can be obtained from the official component registry.

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #1
How to use this:
Code: [Select]
var raw = fb.TitleFormat('[%played_times_js%]').Eval();
var playedTimes = [];
try {
playedTimes = JSON.parse(raw); // this is required because the value is a string
} catch (e) {
fb.trace('<<< ERROR parsing JSON >>>');  // you probably don't need this try/catch, but it was helpful in my debugging
}
for (i=0; i < playedTimes.length; i++) {
var p = new Date(playedTimes[i]);
// do something with this value
}

Just as an example of what you can do, I'm taking the individual play times and plotting them on a song "lifecycle" timeline where it starts at the date the song was added, and runs to the present. I calculate each play's "age" by subtracting it from the current time (new Date() ), and then use the ratio of age / added to determine where to draw the line.

For the record, you can convert a Foobar string timestamp ("2018-01-04 00:36:15") into a string you can pass into new Date() by doing the following:
Code: [Select]
var added = fb.TitleFormat('%added%').Eval().replace(' ','T');

You can compare lastfm dates to dates provided from foo_playcount using the following code:
Code: [Select]
$ifgreater(%lastfm_last_played%,%last_played%,%lastfm_last_played%,%last_played%)

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #2
CHANGELOG

v2.1.1 - 2018-04-11
 - No longer escaping usernames with underscores in them

v2.1.0 - 2018-04-07
 - Added three new properties:
%lastfm_added%
%lastfm_first_played%
%lastfm_last_played%
 - All return a date/time string like "2018-03-04 12:34:56" exactly the same as their counterparts from foo_playcount. If the time doesn't exist, "N/A" is returned, but $if2, or the bracket format ([%lastfm_added%]) will not display the N/A.
 - %lastfm_added% and %lastfm_first_played% both return the exact same value, but I created both for convenience. If you'd like to use the most recent date between your foobar library and last.fm you can use this TF code:

v2.0.4 - 2018-03-27
 - Fixed issue where if "Compare Album fields" was unchecked but the album field was missing in the file tag, the component was not pulling scrobbles.

v2.0.3 - 2018-03-01
 - Fixed crash in foobar if artist/album/title tags were missing from media and checking scrobbles.

v2.0.2 - 2018-02-24
 - Add "compare album fields" option in preferences. Disabling this will not consider album titles when checking scrobbles.

v2.0.0
 - Now retrieving scrobbles from last.fm
 - Right click-menu options
 - New preferences page

v1.0.2
 - Changed name to foo_enhanced_playcount

v.1.0.1
 - Updated the component so that it returns an empty JSON array if no playtimes have been saved so far.

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #3
So, I know I said I wouldn't query last.fm, but now I'm considering querying last.fm when we're about to record a new play. When saving this data to the component's index db, I plan to remove all plays that the component already knows about. That means this would only record last.fm plays with a timestamp the component didn't record or can't find from %first_played%/%last_played%.

There are some issues with this, namely that last.fm does not differentiate beyond "artist - album - tracktitle" where foo_playcount and this component also use %date% %discnumber% and %tracknumber% to differentiate songs.

For example:
I've played the song "Battery" from the original Master of Puppets release in 1986 five times.
I just got the 10 disc Master of Puppets reissue which came out in 2017. Last.fm will report plays of "Battery" from disc 1 and also the 1986 release.
Even worse, Battery appears 4 other times in different live recordings on the 10 disc reissue. Because my tagging does not separate out all those discs (i.e. for one of the live discs, album is tagged "Master of Puppets" and not "Live at Solnahallen, Stockholm, Sweden - September 26th, 1986") if I play "Battery" from that live disc, it will also count as a play of the song on the 1986 release, which is very obviously wrong.

last.fm does include MBID's for releases and recordings, but in my testing they are very often wildly wrong (when they are even present) and it seems every version of "Battery" will map to the same MBID, regardless of whether they really are different or not.

Is this a limitation people are willing to live with? Does a system that works perfectly 98% of the time, but drastically overcounts plays the other 2% of the time okay? Should last.fm plays be mixed in with the standard calls, or separated from the other plays (i.e. adding a %played_times_lastfm_js% field)?

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #4
Is this a limitation people are willing to live with?
Yes. Absolutely. Either in this component or another separate playcount component.

And in case you missed it,Last.fm is planning on revamping their api in the coming months according to the devs posting on the getsatisfaction.com/lastfm board. What that means at this point I couldn't tell you, but hopefully they'll add some new endpoints. Thanks.

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #5
Is this a limitation people are willing to live with? Does a system that works perfectly 98% of the time, but drastically overcounts plays the other 2% of the time okay? Should last.fm plays be mixed in with the standard calls, or separated from the other plays (i.e. adding a %played_times_lastfm_js% field)?

I'd say no. This would be a reason for me to avoid it completely. Most scrobbles on last.fm that are not from the user itself come from either youtube or spotify and spotify in particular is a complete mess with its tagging. The more someone uses other sources to scrobble which they didn't tag themselves the more there will be a disconnection between the "real" plays and the ones reported. What loz is saying about lastfm revamping their api in the coming months should be taken with a serious grain of salt as the site has been in beta since september 2015 and new implementations are moving on with a snail pace (and getting slower).

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #6
I'd say no. This would be a reason for me to avoid it completely. Most scrobbles on last.fm that are not from the user itself come from either youtube or spotify and spotify in particular is a complete mess with its tagging. The more someone uses other sources to scrobble which they didn't tag themselves the more there will be a disconnection between the "real" plays and the ones reported. What loz is saying about lastfm revamping their api in the coming months should be taken with a serious grain of salt as the site has been in beta since september 2015 and new implementations are moving on with a snail pace (and getting slower).
Yeah, I'm not holding my breath on the API changing anytime soon.

You're actually bringing up a second issue. I was only referencing the first where an album with the same song on multiple tracks (live versions/demos etc.) could show up all together (assuming the song isn't tagged differently for whatever reason). That's the overcounting problem.

When you're scrobbling songs you haven't tagged, then there's a possibility it won't show up in the list. I tag using an edition field because I don't want the album title to be "Master of Puppets [2017 Deluxe Remaster]". If Spotify or whoever has that as the album title then when I look for plays with an album exactly matching "Master of Puppets" then it won't show up. That's the undercounting problem and one that I can happily live with. Sorry it didn't count the one time you played it on YT from some guy's sketchy upload.

Now, maybe that truly is a dealbreaker, but you'll easily be able to stop pulling last.fm plays just by not entering in your Username.

That said I think you're pushing me towards having a separate blob in the index_data which would keep last.fm plays from getting intermixed with the foobar recorded plays. Not sure if that means having a checkbox to check when you want it to return last.fm plays in the list as well, or having just another field which will return those and allow the user to dedupe as needed.

 

Re: foo_enhanced_playcount - Record every time a song plays (JScript/WSH)

Reply #7
v.1.0.2 release notes
 - File name change from foo_enhanced_playback_statistics to foo_enhanced_playcount
 - In preparation for retrieving last.fm playtimes, js playtimes now return timestamps rounded to the nearest second.

Note: If you have a version installed prior to v.1.0.2, you will have to manually delete it from your components folder! The filename has changed from foo_enhanced_playback_statistics to foo_enhanced_playcount. Foobar will fail to launch if both are installed at the same time. Please delete the entire foo_enhanced_playback_statistics folder.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #8
I've spent the last moth hard at work on last.fm support. I'm happy to say that it's finally in a state where I'm comfortable releasing a beta version to the public. This has not been an easy project, mainly because last.fm kind of sucks. Also, if you haven't switched to using foo_scrobble from foo_audioscrobbler, you ought to do that immediately. A note on terminology, I use "plays" to refer to a recorded play (from foobar or last.fm) that the component knows about and "scrobble" to refer to a play in last.fm.

Here's the skinny. New fields:
  • %lastfm_played_times% - Date formatted list: ["2012-08-04 15:58:37", "2012-12-10 14:40:46", "2018-01-02 23:38:13"]
  • %lastfm_played_times_js% - JS timestamp list: [1344117517000, 1355172047000, 1514957893431]
  • %lastfm_play_count% - Count of last.fm plays, a la %play_count%: 5

New functionality:
Two new right-click menu options, "clear saved last-fm plays" and "retrieve last.fm scrobbles." You cannot retrieve scrobbles for more than 50 songs at once. Also, 50 is probably a really bad idea unless they're all from the same artist. More on that in a bit.

There's a new preferences pane with a bunch of complicated preferences on it. The reasons and explanations for these preferences are in nifty tooltips. Read them and ask questions. Last.fm scrobbling will not work unless you first go to preferences and enter your last.fm username.

Now the bad news. Last.fm's values for artist/album/title often suck (particularly if you're using foo_audioscrobbler... seriously switch to foo_scrobble immediately). This component requires exact but case-insensitive matches between all of these fields to know whether to count a scrobble last.fm reports as being one for the current track. If there's the slightest mismatch, it won't be counted. This component currently makes no assumptions or guesses. Example: If you are checking for plays from Kesha, last.fm will return them all with the artist named "Ke$ha". %lastfm_play_count% will always show 0. Sorry. Edit: I no longer compare artist strings as I already queried last.fm with the artist. If album and title don't match you'll still have problems though.

If you tag according to musicbrainz and are using foo_scrobble you should typically have a high rate of success though. I've pulled scrobbles for well over 1000 tracks at this point, and I would say 95% returned values correctly.

Unfortunately, Last.fm is slow. I must query by Artist, and can only pull 200 scrobbles at a time. For the first last.fm check of a song, I will pull up to 1000 scrobbles (5 API calls) and then attempt to match them. For all subsequent calls for songs by that artist the API responses are placed in an LRU cache, meaning they should be essentially instantaneous. By default I store up to 20 API responses, although this number can be increased to 50 in the preferences.

At the moment Last.fm calls are blocking on the UI. Yes, I realize this sucks.

I should also say that processing any of the fields this component provides are probably slower than typical. I have not noticed any issues with this, but you might see odd things happen if you add one or more of these fields to a column and try and display 1000 items. This isn't a use case I've tried, so that's why this is still considered beta.

All feedback welcome.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #9
Todo list:
  • Caching currently only works at most when the song has never been scrobbled, and then again after first scrobble. Subsequent calls restrict the API call to after the last known recorded last.fm play. This means these API calls aren't cached. I need to be smarter about this. Done
  • Thinking about allowing the component to use user-settable fields to match against. Something like LASTFM_ALBUM_NAME and then you could tag affected files with that field, e.g. set the value "My Album" and it would fallback to that if a match could not be made for %album%.
  • Ignoring files with a LASTFM_NO_SCROBBLES meta tag because pulling scrobbles incorrectly matches other songs
  • Last.fm calls ought to be done in a separate thread. This is hard and I'll probably save it for last because I'm not a C++ developer and I hate this shit.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #10
Releasing beta 4. Realized after typing up all that stuff that I have no reason to compare artist with last.fm's artist string. We're querying for artist already, so if last.fm returns results it's got to be for that artist.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #11
Beta 5:
  • Updated caching system so that ALL calls to last.fm are cached, regardless of whether a song already has recorded last.fm plays
  • Cached responses now expire if the cache is full and the response was the least recently used page OR this song has already read the current response page from the cache once. In the second case I immediately re-query the API, and clear the list of songs that have read from the cached response page (since it's new).
  • Improved console logging. Will reduce the amount for actual release.
These new changes did break the reporting of the amount of memory the cache is consuming. Not sure I can fix it ever, and not sure it really matters that much since we're really only talking about 7.5MBs of memory absolute worst case (50 cached pages, each the max 200 songs for ~150kB each).

This was a pretty substantial change, so please report any issues you find.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #12
v2.0.0 has officially been released. See the first post or get it from the components page:

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #13
I'm a bit puzzled as to the use of %lastfm_play_count%, is this supposed to display the scrobble count of the song retrieved from last.fm? If yes, under what circumstances would that be the case because I don't seem to notice much consistency in what is shown. At what point is lastfm queried? At the start? When the song gets scrobbled?  I would expect a song that has been played and scrobbled, then restarted to display %lastfm_play_count% as 1 but it doesn't, then again if I replay a song I played and scrobbled from the day before %lastfm_play_count% shows the correct amount of plays.
I also have noticed for example a song which has 1 playcount and one scrobble starts out as 1 displayed local play count and 0 displayed %lastfm_play_count%. It gets scrobbled at the halfway point, local play count increments to 2 and then suddenly %lastfm_play_count% turns from 0 to 2.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #14
Hi, if I install foo_enhanced_playcount, should I remove foo_playcount, or do they coexist?

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #15
foo_playcount really has nothing to do with last.fm.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #16
One other question, is it retrieving the scrobble count of artist-title or artist-album-title?

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #17
Hi, if I install foo_enhanced_playcount, should I remove foo_playcount, or do they coexist?
They coexist. Do NOT remove foo_playcount. It will actually improve your experience with foo_enhanced_playcount (and I don't track %added% or provide %first_played%/%last_played% fields either)

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #18
I'm a bit puzzled as to the use of %lastfm_play_count%, is this supposed to display the scrobble count of the song retrieved from last.fm? If yes, under what circumstances would that be the case because I don't seem to notice much consistency in what is shown. At what point is lastfm queried? At the start? When the song gets scrobbled?  I would expect a song that has been played and scrobbled, then restarted to display %lastfm_play_count% as 1 but it doesn't, then again if I replay a song I played and scrobbled from the day before %lastfm_play_count% shows the correct amount of plays.
I also have noticed for example a song which has 1 playcount and one scrobble starts out as 1 displayed local play count and 0 displayed %lastfm_play_count%. It gets scrobbled at the halfway point, local play count increments to 2 and then suddenly %lastfm_play_count% turns from 0 to 2.
The component retrieves last.fm plays when foobar marks a song as played, i.e. 60 seconds in. This component does NOT scrobble, and scrobbles happen after a song is played. So, %lastfm_play_count% will always be 1 behind the %play_count% if you don't manually pull scrobbles from the right click menu.

There's an option in the preferences which will attempt to keep %lastfm_play_count% in sync with %play_count% by adding 1 to the total if the last foobar play is later than the last scrobble we have recorded. This +1 only happens if we have at least one scrobble recorded. So the first time you play a song %play_count% goes to 1, we see no scrobbles and leave lastfm_play_count at 0 (because I can't differentiate between never scrobbled and never checked for scrobbles). The 2nd time you play a song, play_count goes to 2, we see 1 scrobble (the previous one because again the current play hasn't scrobbled) and then we add 1 because we assume the song is about to be scrobbled.

I can play around with this logic, or figure out a way to mark whether we've already checked and the song just has zero scrobbles.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #19
One other question, is it retrieving the scrobble count of artist-title or artist-album-title?
Artist-album-title, and remember it requires an exact (case-insensitive) match. I can play around with that test if you can show me songs you've scrobbled that aren't showing up. The component logs the last.fm API calls it makes so you can see them yourself and determine where potential mismatches are.

If you're diligent with tagging songs and albums correctly, and you're using foo_scrobble to record scrobbles it works >99% of the time in my experience.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #20
Thanks for the explanation. The artist-album-title makes sense and clarifies the play count -vs scrobble discrepancies I was seeing. I was working under the assumption artist-title. It's too bad really it's not artist-title based because spotify with its louzy album naming throws a wrench into making my foobar plays and scrobbles be more in sync.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #21
Uploaded v2.0.1 but it was just a version bump so update check could differentiate from the 2.0.0betas.

Thanks for the explanation. The artist-album-title makes sense and clarifies the play count -vs scrobble discrepancies I was seeing. I was working under the assumption artist-title. It's too bad really it's not artist-title based because spotify with its louzy album naming throws a wrench into making my foobar plays and scrobbles be more in sync.
I think I'll add a property in the dialog that would disable album title checking (disabled by default). Would you be cool with that? Would still require exact title match, and I won't change that.
Pros: Your spotify scrobbles would get counted more reliably.
Cons: Every play of a song from a live album, or single, or whatever would all get lumped together into one.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #22
Disable album title checking would be great. The con is a non issue to me since I already lump them together anyway.

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #23
Disable album title checking would be great. The con is a non issue to me since I already lump them together anyway.
Update should pull v2.0.2 with the new setting. Can you go into preferences and verify that the Compare Album setting is checked for you by default? Uncheck it and your problem should be gone. Thanks for the idea!

Re: foo_enhanced_playcount - Record all song plays and Last.fm scrobbles (JScript)

Reply #24
Album settings is indeed checked by default. After unchecking it %lastfm_play_count% is what I'm expecting it to be. Thanks for the update.

 
SimplePortal 1.0.0 RC1 © 2008-2018