HydrogenAudio

Hosted Forums => foobar2000 => Development - (fb2k) => Topic started by: regor on 2021-01-05 21:54:43

Title: [SMP] Music Graph development
Post by: regor on 2021-01-05 21:54:43
First my request: I'm looking for a few users to try and test a SMP collection of scrips I have been working on. When they are ready, I will public them on github and here, but right now I'm just looking for betatesting. This is my first time coding on JS and after +2000 lines of code some help would be appreciated testing it!

Back to the topic. Some time ago, while tinkering with foobar I found it was really difficult to create autplaylist based on similarity checks. Things like these:

Things like Random Pools component, current SMP scripts or Search for same components (I know 2) have many limitations for those things:
X


I could continue...  but essentially, foobar is missing the functionality spottily gives, or allmusic, or musicbee, etc. Essentially the ability to create playlist dynamically based on tags or internet info of the currently selected track. There are some things done on that field but really limited on functionality.

There is also a music server app name MusicIp (https://www.spicefly.com/article.php?page=what-is-musicip) which works similarly to the things mentioned. Short story: it requires days to scan 50k tracks, it's an unsupported program from 10 years ago,  requires the server to be loaded along foobar, CMD scripts to send tracks to server and load playlists AND... it doesn't support multi tags. So all auto-playlist created rely on the musical analysis, but tags become useless. Also rating is not loaded, you have to create scripts to load foobar ratings to the server, manually! The program accepts some scripting and advanced commands, but there is no documentation and it's closed source... so good luck with it. Btw, someday I will post the things I managed to make work properly (only thing missing is multi-tag support).

All these findings lead me to AI reasearch. I mean... MusicIp "works" but it uses musical analysis from a decade ago. Nowadays we have machine learning models giving us things like genre or moods. That's the thing Spotify, Allmusic or any other network player uses. So we have 2 problems to solve: first, finding how to analyze music using modern methods. And tag it! Second, use foobar to create dynamic playlist with that info!

The first one is partially solved using Musicbrainz Picard (https://picard.musicbrainz.org/). You can get mood, BPM and key tags from that program. And any file missing those variables can be analyzed with an exe (https://acousticbrainz.org/download) which uploads the analysis to the server; 1 day later you re-load those files in Picard and magic. The missing tags are there.
I have managed to tag  (mood, occasion, BPM, Key) 99% of my files with that method, while adding manually things like genre and style ("subgenre"). Things like similar artists are provided by the last.fm component (can be saved to local tags!) or SMP scripts.

Tagging is mostly done. but the second one, previously discussed, its unresolved up to now. We have 2 things to consider now: dynamic playlist creating and genre similarity, whatever that means. Back to AI research, I thought I could find something useful on that field... without luck. "Solutions" require complicate scripting, a range of tools and then finding a good set of reference... but someday I found this Music map (https://musicmap.info/). There have been other similar maps, most interactive, giving you youtube videos of the selected style.
X


So, considering automated music genre mapping was out of question, I thought that could be translated to code. At the same time I was beginning to discover SMP scripts, so I tried to translate that map to js. This is my fist time on js, so it has been a hard road.

My first attempt is what I call "DYNGENRE". It's a pretty simple concept: let those 18 Supergenres (following the web convention) be assigned a number from 0 to 18. Any genre tag or style tag should fall into one of those supergenres. We can simplify it even more and not make any difference between "modal jazz" or "cool Jazz", they are just jazz (9)! And assign to those styles which are a mix of some supergenres their set of numbers: "Trip-Hop" (12,18) would be "Hip-Hop" (12) and "Downtempo" (18).

That assigns any track to a Supergenre (or a mix) according to genre tags and styles, and the best part: it's done on the fly. No need to re-tag. No need to recheck files when changing scripts. The DYNGENRE map can be set as you wish just by tweaking the js file.

The other problem was dynamic playlist creation, but I managed to code that for the currently selected track. It takes the mood, BPM, Genre, Style, Date and key tags and create playlists based on those tags. That means no more statics pools. You select rock from the 90s, and it outputs things according to that directly to a playlist (bypassing the need to press 20 buttons using the "search for same" plugins).

While working on that, I got the "duplication tracks" problem again... essentially, if I looked for things similar to a  Led Zeppelin track then it would output things from Led Zep,  Deep Purple, etc. but If I had 4 versions of "You shook me"... then all 4 got selected on the final playlist! That was one of the biggest problems with other plugins too. I want a playlist with things appearing once, not the same track all around from different albums, compilations, live sets, etc. So I created some scripts to solve that too (https://hydrogenaud.io/index.php?topic=116669.msg991881#msg991881).

The "remove duplicates script" looks for tracks on the handle list or playlist with same title/artist/date and remove all but one. It can be configurable to any other tag too and maintains the order. The last part is important! The script greatly surpasses the built in foobar remove duplicates functionality and the one on SMP methods too.

Other things I added are the combination of multitags. i.e. "I want only tracks with 5 similar moods", but if the tracks have 8 different moods, then you have to create k-combinations (k=5) of the entire set and check for every combinantion via queries. That's impossible to do on static pools or manually. Then we have user adjustable weighting of tags, filtering of any track below a minimum score, pool selection by lowest distance or random, etc
X
 
With all the problems solved, the final result is a playlist generated based on the tags of currently selected track with no duplicates and following similarity rules. All user adjustable.

And that's the "DYNGENRE" method which would be a more complex version than the "WEIGHT" method. My collection of scripts, ready to use, offer that (search by distance), along remove duplicate scripts (ready to be used standalone on playlist), Top 25 Tracks (without the duplicates problem), search similar style + k-com of moods and Search for same style (multitag).

This is a screenshot of the thing working. I took a folk track, and 857 tracks were eligible on the pool (score over 65%). Final selection, 50 tracks.
X
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-05 22:22:28
After all this discussion, here it comes the second part. Along the "DYNGENRE" and "WEIGHT" methods, there is a third one: "GRAPH". Remember the previous music map examples... I simplified the thing assigning main genres/styles to numbers and not much more! But what about a real map (graph) with genres and styles? This:
X

Well, after coding all the previous things I tried creating a real map. With distances and links. I found this js library (https://github.com/anvaka/ngraph.path) which does exactly what I wanted: graphs, nodes, links and the most important thing, pathsearch. But it was for node.js. I find how the hell put that thing to work within foobar and got a graph created using arrays with all the genres and styles. And I mean all. Then adding supergenres, clusters, style clusters, etc.
X

I put all that on the "search by distance" script, configured some limits for the paths and got it working. Now "Coutry Blues" is not just "Blues" (dyngenre method), but a node within blues supergenre. And it would be more distant to Electric blues than to Delta blues, for ex. Then we got style clusters and primary origins/derivations to traverse between different super-genres.. and all that. It takes 2-5 seg to search 40k tracks and calculate paths and distances within foobar. It works, and creates playlists.

My final step on the development have been the graphical part. Coding +60 arrays with nodes and links is great but it becomes a crazy thing, to the point I don't know if I'm duplicating things or setting things wrong. And here the graphical representation comes to help. I managed to render the graph within firefox using other JS library (https://github.com/anvaka/VivaGraphJS) and creating an HTML file pointing to the foobar js scripts. Whenever you edit something within foobar scripts, it gets automatically loaded on the html. And that's the first image, you can zoom, move, hover the nodes and look for the IDs for debugging. With that tool I have managed to solve some problems I had at the graph today and added Folk and Classical clusters too, so that should make all popular music up to date (but the "World music" at musicmap web).

For that part, I would gladly accept some help rendering the map according to link weighting (i.e. the real precomputed distance) instead of the viva methods by link centrality. That goes beyond my pretty limited knowledge of js and would be the final piece.
Title: Re: [SMP] Music Graph development
Post by: mjm716 on 2021-01-06 02:05:57
First my request: I'm looking for a few users to try and test a SMP collection of scrips I have been working on.

I wasted a bunch of time with MusicIP back when it was beta - a lot of potential, as it generated the best related mixes I ever encountered. However it had the slowest track scanning ever and proprietary model that rendered it quickly useless.

I'm interested in yours - count me in. How?

It sounds great and also like it could lead to some personally long wanted FB statistical functions that have been out of my depth so far.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-06 20:56:21
Oh thanks! Just check your pm inbox in a few days ;)

This one generates mixes as good or better than musicIp but you need to have variety on your library. i.e. if you only have 20 folk tracks, then "folky playlists" will be really limited... while musicIp didn't care and filled the gaps with anything until it filled the minimum playlist length. The good part is that anything is configurable here, so you can lower the score filtering too...

This script also can be set to just output similar style/genre tracks or by mood or whatever, so you can also emulate those musicIp remixes which had different genres but "natural" variations.

I would want to recreate dynamic variations too. MusicIp was able to use one track as reference, output some similar tracks, use one of them as new reference for the rest of the playlist and so on. Also I could add more control to the content of the playlist, like X 80% similar tracks, Y 50% similar tracks, etc.Currently it's totally random, by score order (only higher scored tracks get to playlists), or by score order with random probability of being picked (it starts at high scored tracks but there is some randomness). MusicIp had so many details that I could work on this during so much time.

I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure (https://hydrogenaud.io/index.php?topic=65185.msg813634#msg813634) the fp I would be so grateful!

I'm currently polishing a few things so I can export all the files to you or anyone. Also managed to put custom images or shapes as nodes on rendering.
X

I have to create some buttons too (since I'm currently using the scripts as  "File\SMP" menus) and I will be ready for betatesting.
Title: Re: [SMP] Music Graph development
Post by: Daeron on 2021-01-06 21:15:53
I feel like a lot of this could be simplifed/made redundant by just smart tagging and understanding how we tend to categorize things in our heads.

I defaulted to just tagging common elements in music instead of following a rigid tree branching structure that people tend to follow. Each artist just has a bunch of stickers that range from very broad to oddly specific. The specificity is increased in areas of your library where you have the most knowledge/gathered the most amount of songs. And the only purpose is to help you quickly gather any subset of artists that have something in common.

Example stickers, ranging from more broad to oddly specific:

Basic genre of the song:
- This is always filled out as your starting point
- This is just to get the most basic idea of how the music sounds like.
- Lots of artists don't fit just one genre, so I often tag them with 1-3.

Subgenres equivalent:
- This is for areas of your library that has grown significantly and you need to break them down into smaller groups
- This requires that you actually learned to spot the small nuances of a genre and you actually care about the distictions
- If all "Electronic" music sounds the same to you, there's no point in meticulously tagging its subgenres
- But if "Melodic" or "Gothic" Metal has meaning to you, those subset of artists are worth tagging more precisely
- More generic terms tend to work better. E.g. "Melodic" can refer to Metal, or Rock, or even Classical music, or Industrial can refer to Rock, Metal or even Electronic music (if you approach it in a specific way).
- This way you tend to get a lot of cross-pollination between genres when building playlists

Moods/themes/feelings:
- This is fairly personal choice. It's more like a twist or "angle of attack" to further differentiate tracks within the same genre and subgenre.
- Words like "Modern", "Bleak", "Upbeat", "Downtempo/minimal" come to my mind. These are similar to subgenres but I'd say more abstract.
- They are more easily repeated across a wider arrays of genres/subgenres that don't otherwise have much to do with each other (you can have an upbeat song from any Electronic, Rock, Pop or even Metal song etc.)

Instruments used/location specific
- If you want you can specifically mark artists or song that emphasize an instrument or lack of
- For example music with standout drum usage (regardless of whether it's EDM, Classical or Metal music)
- Territory specific music that tend to use a similar set of instruments/style of music (E.g. Arabic or Asian)
- Instrumental songs vs songs with Female or Male vocal lead

Easy listening:
- This is to simply mark songs that are specifically easy, or hard to listen to, to filter potentially problematic songs from your autoplaylists
- Tracks don't tagged are counted as neutral

With this kind of tagging you can quickly get different kind of sets of music (either very broad, or very specific), or use them to generate nice autoplaylists with not that many rules in the query:
- All electronic music
- Only Melodic Metal music
- All gothic sounding music (ranging from Pop, to Metal, to Rock, to even Classical music)
- Electronic or Rock music that has no vocals but is generally upbeat
- Film scores that sound "Modern" and has a heavy emphasis on drums
- Anything that has an "Industrial" sound to it
- Japanese Pop music with female vocals
- etc.



Apart from these I also tend to use ratings in two ways:

Ratings supplied by the listener:
- People are not very good at rating objectively in a capped system (e.g. people usually rate from 7 to 10 in a 10 based system, unless the track is completely terrible)
- People's taste changes over time so what was rated 5/5 might not be subjectively thought as a 5/5 later
- In my system you can press a button to increase the songs rating by 1.
- Any song that's 1 or above is considered a favorite and can be included in autoplaylists
- Songs that are rated below 0 are banned and won't be part of normal listening
- Ratings don't have an upper cap so it's a numbers race. After sufficient amount of time passed you get a few tracks that are rated absurdly high and you can make queries based on this to filter out the "best" songs

Ratings tracked by the computer and the listener can't tamper with:
- Using the DAR system (https://hydrogenaud.io/index.php?topic=83954.0) for this
- People tend to cheat with the ratings and artifically inflate them, if they get the choice (such as pressing the button multiple times)
- This represents the real time listened vs the subjective rating
- Using both numbers you can reasonably estimate the actual top songs
- Or just gradually phase out songs that have been listened to a lot, or force yourself to listen to those that got not much playback time



With the combination of the two things above you can make really well personalized sets of tracks that fit your current mood really well (via Facets or CUI Filter Panels).

My only gripe is that the random pools component (which has an option to basically normalize how often an artist's songs are presented) doesn't really let you save different presets of finely-tuned playlist generation rules, or that foobar can't really keep a running tally of "this artist has been played before in the last 20 minutes", to give equal chance to artists who only have a couple tracks.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-07 00:21:15
Btw it's already done and working on my side so there is no need to simplify things; I'm just looking for people to test it.

Your approach is the base of the things I'm discussing, but not the end. Yes, you can manually create playlists if you have tags set like that. That's what I have done on my library and it's what it's required for the scripts to work!

But that's not the point. I have my library tagged exactly like your description and I miss tons of things. Specially creating playlists dynamically. If I told you to select tracks similar in mood to the current one, you could spend 10 minutes creating combinations of moods...What is similar? 1 mood? 2? 5? Then the same with genres, with styles, etc. You know British Folk is similar to Folk Baroque, but .. that's in your head. Foobar knows nothing about "genres". You have to put that in tags in some way.

Autoplaylists, facets and all that have many weak points:
-Require direct user interaction. Queries become unmanageable for complex things. The user must check every playlist created to be sure they are right (duplications, ratings, etc.).
-They can not be changed dynamically. Yep, you can create a query for rock with moods "acoustic, sad, agressive, blabla", But then if you want the same for Folk, from specific dates you have to recreated the entire thing manually
-It's impossible for an human being to translate combinations of tags into queries manually. Check this image with a 15 lines query: https://hydrogenaud.io/index.php?action=dlattach;attach=18631;image
-They can not be set according to a source track. Yep, you can look for Rock or Glam Rock and all that. But you can not create an autoplaylist or Facets filter with "x same tags" to the current selected track.
-Can not be saved. Facets solves that partially.
-Can not filter tracks. You apply a query and that's all. Forget about filtering duplicates or other similar things. In fact, queries are created -by definition- to give you all the tracks found. Duplicate detection is out of the equation. I have many versions of the same songs, I don't want to listen to an autoplaylist with the same track 5 times.  "Top 25 tracks" playlist is a clear example of this problem.
-Exclusions are always manually set: Heavy Metal but not Black Metal and Folk Metal is OK but Folk Metal and Folk-Rock is not, and...
-Can not set playlist lenght.
-Require a crazy amount of work to make it work properly. Yes, you can add more and more and more tags for corner cases and complicate queries... but it makes no sense at some point. I want a system which identifies folk sub-genres and their relations, not having to add a trillion of tags to differentiate them (Instruments used/location specific & Easy listening:); work is done once, etc.
-What about the day you just want to listen to music and not to spend time with all those silly things?

Essentially it's simple. Foobar already has all the ingredients to do things like modern music players, but ironically they have not been mixed yet. Random pools is an example of that, it's development is against that aim of fine-tuning playlists. For ex. it has already been asked to assign pools to playlists names, but that request has been rejected due to being "too complicated".

Even foobar, which already has many title-format variables, omits the current selected track or playing track data. If foobar gives us %currentplaying_genre" or things like that, it would be a joy to create dynamic auto-playlists or random pools according to the selected track. You can not create playlists using other playlist as source, etc. Check MusicBee for those details missing.

We are at a point where machine learning, automatic tagging or music recognition are "simple" and userfriendly, so what I expect about a modern player is applying those concepts locally. I don't like Spotify because it only works via internet. Do you remember Genius from itunes? The same. I want the same things found on those players but working on my PC. I don't want to learn 8000 queries and tags to "teach" foobar to make playlists, I want foobar to show me my local music the same other players do... finding related things automatically. And in the process, to offer many more things.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-08 23:37:26
Just finished making buttons to test it easily. Also for the other functions, with hover text, and ui changing according to preferences.
XX
XX
Quote
For that part, I would gladly accept some help rendering the map according to link weighting (i.e. the real precomputed distance) instead of the viva methods by link centrality. That goes beyond my pretty limited knowledge of js and would be the final piece.
And somehow I managed to solve that.
X

Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
But that still applies.

Sent you a PM mjm716
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-14 17:07:30
Development continues and I have added some new things to the set of scripts.

-Buttons framework to merge/add/create new buttons seamlessly to any SMP bar. That means... you create a button file for whatever you want (sort the playlist for example) and then you simply "include it" on a main file. The main buttons file loads all included buttons, sets position, etc. Zero coding (and no need to edit the other files to merge them). Enabling or disabling buttons just takes 1 line.
X

-Included some example buttons. An user configurable button within foobar (using properties panel) and 2 new buttons with new functionality (some auto-tagging processing and playlist filtering).

-Updated the HTML rendering file with a few things.... it precomputes the graph before rendering it, pins the node info, etc. Font and label sizes are now set according to window size.
X
X

-Added extensive comments over the scripts.

-Added a remove duplicates function which can be used as playlist filtering, setting how many duplicates are allowed... you can filter by date and only allow 3 tracks per date or things like that. And speed improvements (thanks to snotlicker)

Betatesting still required! Thanks :)

Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
And help required for that yet!
Title: Re: [SMP] Music Graph development
Post by: TheQwertiest on 2021-01-14 23:26:59
Wow! That's really impressive!
I could not even imagine that such cool stuff would be performed via my component!
Keep it up! ;)
Title: Re: [SMP] Music Graph development
Post by: zeremy on 2021-01-15 14:08:22
Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
And help required for that yet!

You would need raw output of fpcalc for the files.

These links may help !

https://groups.google.com/g/acoustid/c/C3EHIkZVpZI?pli=1 (https://groups.google.com/g/acoustid/c/C3EHIkZVpZI?pli=1)

https://gist.github.com/hertzg/c75aa10072d10f00b38e (https://gist.github.com/hertzg/c75aa10072d10f00b38e)
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-15 20:17:54
Wow! That's really impressive!
I could not even imagine that such cool stuff would be performed via my component!
Keep it up! ;)
Thanks! Well the html rendering part is done outside foobar with a browser, so it's just using standard js code. I'm not sure if it could be integrated in any way within foobar as a html popup though (?) On github, you told me IE is used for html rendering, so I could try that path once I install it. I don't use fancy things, just standard js, SVG rendering and jquery as a library. It would be awesome: I could do things like showing tracks from the library matching the node selected, playlists by node and "radius", etc.

The graph creation is done with a node.js library, but works within foobar and browsers; I simply had to change a few things on the ngraph library to make it work.

The distance calcs and playlist and all that is done within foobar, your component is fantastic I think any functionality from modern web players can be replicated locally without problems with js. This is just an example. The only limitation I have found is processing time. Spotify has tons of servers calculating these same things! It takes some seconds on our PCs to create a graph or distance calculations for all the library. I even think machine learning models can be paired with your component for auto-tagging or things like that. We just have to see js as a door which allows us to load arbitrary things within foobar and manipulate the library and the playlist with all that info!

Quote
You would need raw output of fpcalc for the files.

These links may help !

https://groups.google.com/g/acoustid/c/C3EHIkZVpZI?pli=1

https://gist.github.com/hertzg/c75aa10072d10f00b38e

Let's see if I find a way to calc the raw fp within foobar (so users don't have to analyze things with other programs*) and the comparison logic. Thanks! I will take a look at that.

*Foobar has a component but uses other fp method. Ideally, I would support both.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-15 22:30:05
Some advances:

-Managed to create buttons bar as easy as just include the button file. 1 line.

-Buttons bar can be merged too -> You just include the button bar path, and it's merged the same than standalone buttons.

-Buttons and bars can be duplicated just by including them twice. This is important since they act like different instances, with their own properties at the panel (they get duplicated with prefixes), variables and all that! (other implementations required iDs to be different, properties description adjusted before merging, creating new files for "copies", etc.)

-All these things mixed means... you can just copy the "include" line of a button example 10 times and configure each button within foobar (properties panel) as you wish. In other words, you can create customized buttons using a predefined template without coding a line. You only need to copy/paste paths at the js file and the rest is adjusted on the fly.

X

X

X

X
In this example, every button does the "search by distance" thing explained on the thread according to different user set variables. That means I don't have to create a predefined set of buttons which creates playlists according to my liking... but the user can create their own in a few seconds.

The great part of the framework is that it's universal. It can be applied to anything, not just my intended use.

-"Similar by" arbitrary tags / number of coincidences. If date is used, then it checks by range instead of number of coincidences. It's a generalization of "search same" functions, user can put whichever tag they want and number of coincidences required for the query. Using it, creates a dynamic query according to the current selected track, and creates a playlist without duplicates (using my remove duplicates function) with random entries from that query. It would be a query equivalent to the "search by distance" implementation. Obviously the query is limited to "equal tags" and simple things while the other uses a graph to find relations between genres/styles and more complex things...

X
Tooltip is adjusted according to the panel properties and tags set.
X
Query example created.
X
Properties panel.




Title: Re: [SMP] Music Graph development
Post by: regor on 2021-01-25 15:10:01
-Remapping and merging tags feature for similarity checks. So not only you can use your own tag names, but also merge different fields into one before evaluating them. This allows us to use the scripts with any tag scheme without needing to re-tag.
"genre tag" -> genre, allmusic_genre, ...
X

-Added 2 custom tags for checking. There is specific logic for genre, style, date, etc. but you can also add 2 custom tags to be checked by matching (strings) or within a range (numbers) according to their own weights. Why? Let's say some user have it's own "occasion" tag, then it can be used on the search by similarity buttons/functions without changing the code. It's just a placeholder for user set tags, appart from the main ones.
X

-Some speed improvements (even with the tag merge/remap feature added).

-Filter for the final playlist. Apart from removing duplicates (by title-date-artist, not just duplicated files),  also a filter can be set to only allow X tracks per artist, per date, etc. Fully configurable (tags and number of allowed tracks per check).

Work is mostly done on this thing, I'm now working on a playlist manager which doesn't force you to have playlist loaded within foobar, so you only load whatever you want. (like a library viewer / album list for playlists).

Right now if you have 100 playlist, you must have them loaded on the program, cluttering the playlist tabs... well it makes no sense to me. Other playlist managers alleviate the issue, but you still need to have them all loaded and then "explore" the playlists list using the manager instead of the tabs. Tabs become totally useless and unmanageable:
X

My approach is different, you have your list of playlist and you simply load the ones you want as new tabs. That way you have the best of both worlds: playlist tab functionality (without cluttering) and a playlist manager. There is something similar for autoplaylists (by saving queries) on SMP, but it's limited to that: auto-playlists.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-02 16:49:58
Playlist manager is ready to be tested too.  Please drop me a line here: regorxxx@protonmail.com
Note it currently uses a rewritten version of the list panel by marc2003, so the UI is pretty similar right now. I plan to change that once I see the internals are working.

A copy of my notes:
Quote
/*    Playlist Manager v 0.1 02/02/21
   Manager for offline playlists. Shows a virtual list of all playlists files within a configured folder (PlaylistPath).
   Features:
      - Auto-saves changes within foobar to playlists files. (configurable)
      - Automatically updates changes within the tracked folder. (configurable)
      - Lock/unlock playlists (so they are read-only).
      - Automatically locking of native foobar playlists files (.fpl). (configurable)
      - Playlist unique IDs. You can have multiple playlists with same name on the UI and binded to different files. (configurable)
      - Show playlist size on the list. (some limits apply for .fpl playlist files) (configurable)
      - 3 different writable formats. (some limits apply for .pls playlist files) (configurable)
      - RecycleBin: deleting and restoring.
   TODO:
      - Merge with autoplaylists ?
      - Search box
      - List ordering by: Name / size / Virtual folder ("category")
      - Add category data to playlists for searching and ordering
      - Drag n drop to add playlists to folder
      - Better fpl support?
         * Save data to json, load data from json, overwrite only non fpl files
         * Replace 'update playlist' with save file UI for fpl files, when locked
      - Replace 'update playlist' for locked files with popup to overwrite file but maintaining locked status
         
*/
Resume: you have a folder with 20 playlists and these are NOT shown on foobar tabs. You only load the one you want and that's all.
You can treat playlists as files, you can delete them and all that.
If you have auto-update enabled, the folder is tracked... so if you add or remove playlists using the explorer the changes reflect into foobar immediately (the same than album list).
If you have auto-save enabled, then changes to loaded playlists are monitored. Whenever you add tracks, change order, etc. the playlist file will reflect the changes. You can also have locked files (so they can only be updated manually).

Red playlists are locked
X

Right and left buttons menus
X

The playlist format: currently supports m3u8 and m3u (full features), pls (no unique Ids, no locked status) and fpl (only read).
X

I managed to create unique IDs with invisible unicode chars. The (*) is just for identifying them for demonstration purposes. That means you can have playlist with duplicates names on the UI and still be linked to their unique files.
X

And finally the point is these are real files and folders (not saved on a universal json file), so you can have multiple instances of the panel pointing to different sets of playlists:
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-02 21:53:11
This is what I was talking about. The panel title is the folder name.
X
And every folder has its own playlists.
X

It could work as categories with different sub-folders for every one (although I plan to show the folders as a tree at some point).
Finally, the files are ready to be used in any other program. That's the main difference between a json approach and this one. The files are not limited to foobar internal use (neither require additional code).

This is wip, but tooltip shows current foobar playlist name associated, number of tracks, file name, if the playlist is locked for modifications, category and possible tags given to the playlist (for future searching feature)....
X

When I merge it with autplaylists, then the tooltip will display other data, like query (up to some length),  number of tracks which will be generated, etc.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-05 16:01:34
Some updates:
- Different Icons for  empty/non empty playlists. Along the red color for locked playlists.
- Double click on a playlists automatically opens it. If it's already open, it makes it the active playlist
- Double clicking works along single clicking and tooltip.
- Added selection indicator.
- Added playlist count to the header.
- Category and tags fully implemented (it's saved on the playlist files too). Next step is a search box or a dynamic menu generated by those categories/tags.
- Added buttons to sort: the button names, tooltip and functionality  changes when clicking. So there are 2 sort modes and 3 filter modes which you can cycle.

X

X

The biggest improvement is the buttons framework, since I have managed to just reuse the one I created for the buttons bar and make it compatible within arbitrary panels. i.e. buttons can be added to anything without complicating the code just by copying a few lines and reusing the same files. It's portable. My biggest gripe when looking to other UI samples and complete themes is they are heavily dependent on tons of functions and variables to the point is a nightmare to reuse them.

I would want to do the same with a scroll bar and a search box, but not sure how many time it will take.

It's fully functional so if you want to test it please drop me a line here: regorxxx@protonmail.com
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-06 11:44:54
More updates:
- Added autoplaylists to the manager. They can be created, modified, deleted and saved the same than standard playlists.
- Standard playlists are linked to physical files on the tracked folder.
- Autoplaylists are saved to a json file. i.e. every-time the panel is updated, they are merged with the list of physical files.
- Tooltip shows the query and sort for autoplaylists.
- Tooltip shows the number of tracks the query would output (always updated to the current foobar database).
- fpl playlists have their size cached when you load them for the first time on foobar. That size is cached (so it shows in the UI even if there is no way to "read" fpl playlists).
- Added a second filter to show/hide autoplaylists. (thinking about a 3 cyclic filter: only playlists, only autoplaylists, both)
- Added a different color to autoplaylists.

At the current state it contains every functionality of Auto-playlist Manager by marc2003 (https://theqwertiest.github.io/foo_spider_monkey_panel/docs/script_showcase/single_panel_scripts/#auto-playlist-manager) for autoplaylists (and some more).  Since I was using that panel too before creating this (and I have tons of lists there), I could add some helpers to migrate the json autoplaylist file to this manager. And it would be helpful for some people too.

Tooltip for AutoPlaylists:
X

Contextual menu for AutoPlaylists:
X

Contextual menu for standard playlists:
X

Panel menu:
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-09 01:22:43
More updates:

X
- Added a menu entry to import json files created by Auto-playlist Manager by marc2003. Migrating autoplaylist from that panel to this one just takes a matter of seconds.

X
- AutoPlaylist size (*) is updated on json import (so they show the same info than autoplaylist created within this panel).
- AutoPlaylist size is calculated only once at script loading. Can be forced to refresh with 'manual refresh'. This is done to reduce lag, since 1x autoplaylist can easily take a few seconds to compute.
- Current index (selected item) and offset (position within the list when scrolling down) is not reset when updating the panel, applying filter, etc.

(*) The number of tracks the query would output.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-11 16:30:31
I have received some mails, I will answer you all asap with new links as soon as I clean a bit the graph script files I did after these changes:
https://hydrogenaud.io/index.php?topic=120394.msg992987#msg992987
Just to be sure I give everybody the last working version.

Back to the playlist manager, More updates:
-I focused on configuration, some safety checks and coding it in a way it can be easily modified (for adding new filters/sorting methods, unique IDs, etc.). And the properties panel and menus are recreated on script init automatically according to those methods (while saving the previous user variables). Should be easy for anyone with basic knowledge to add their own things there.

-Properties descriptions change according to things set on the panel, not just the values. i.e. if you change the sort method using the contextual menu, then the description on the panel reflects the associated states dynamically.
X

-Sort button can be changed to sort by Name, Category or Size. UI updated accordingly.
X

-Sort state is saved between different sessions/reloads.
-Sort method is saved between different sessions/reloads.
-Filters are saved between different sessions/reloads (configurable).
-Added contextual menu options for:
   -Sort method: sort by Name, Category or Size
   -Show Playlist Size: All (refresh autoplaylists queries), only standard playlist (1), no.
   -UUIDs: invisible unicode chars plus (*), (a-f), (*) or nothing.
   -Filter saving
-All playlist are loaded in cache once, filtering just changes the "painted" playlist on the list. This fixes autoupdate firing when using filters (and other complicate workarounds). Speed improvements.
-If you choose not to refresh autoplaylist sizes (1), then the original size gets used. Original size is always calculated when importen from json or creating the autoplaylist (done once in ther entire lifetime). Greatly speed ups panel loading while maintains some size usability. They can always be refreshed manually though.
-Json file backups periodically on script init/reload. Old file gets copied to .old, so there is always a previous version to restore if something gets broken.
-Modified button framework to allow on the fly coordinates calculation, so it merges the recalc of buttons bar with width and height windows resizing.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-13 14:56:47
More updates: (last ones before I send the new batch of links)
- New updates are delayed when performing internal updates/processing to avoid conflicts when editing files.
- Separators between different names/categories (configurable). Numbers are grouped by #
- Now playing and loaded playlist indicators.
- Selection rectangle indicator.
X

- Background text is wrapped when text width is > than panel width. When showing playlists, the playlist name is trimmed.
X

Code: [Select]
	TODO (by relevance):
- Better fpl support?
* Save data to json, load data from json, overwrite only non fpl files
* Replace 'update playlist' with save file UI for fpl files, when locked
* Native FPL support ? (requires new SMP version)
- UUID:
* nextId() called only for new playlist, reuse old id?
* Reassign UUIDs when changing method?
+ First save loaded playlist and then close them?
+ Rename loaded playlists?
* Save UUID on json but not on files?
- Search box
* By Name
* By tag
* By category
- Drag n drop: (requires new SMP version)
* Add playlists files to folder (requires new SMP version)
* Add tracks to playlist file (requires new SMP version)
- Scrollbar
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-14 23:51:32
Added configurable colors for different elements of the playlist manager and fixed some bugs for the GRAPH files. The main difference is the ability to remap tags and use TF expressions which should cover everyone's use-case.
https://hydrogenaud.io/index.php?topic=120394.msg992987#msg992987

Just sent a mail with a new link to everyone who asked for it.

Next things to do for the graph files is considering adding some configurable tag filtering (like filtering 'soundtrack' when comparing genre values), adding more genre/styles and links to the map and maybe some more complex link logic to account for influences (reducing the total mean distance by an absolute value).

Would love to offer better fpl support (write/reading/updating without user intervention) on the playlist manager, but since it's a closed format and there are no funcs exposed to work with them I can not do anything more. Sorry.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-16 12:42:52
More updates for the Graph scripts:
- Configurable tag filtering (like filtering 'soundtrack' when comparing genre values). So those tags are not considered for scoring purpose.
- New linking logic to account for influences when they happen between 2 nodes within the same SuperGenre (reducing the total mean distance by an absolute value the opposite to how 'anti-influence' works).
- Fixed a small bug on linking between Industrial_cluster, Four-to-the-floor Dance_cluster and Downtempo_cluster when linking in circles.
- Added extensive debug checks when loading graph scripts within foobar or HTML, with popup message and console warnings if there are errors on the graph descriptors:
      +Not connected nodes.
      +Nodes connected to their-selves.
      +Substitutions with influence values != 0.
      +All distance variables to create the graph (according to some logic).
      +Basic path testing (SuperGenres, SuperGenre Clusters and SuperClusters in circles), distance should be always greater than a threshold (according to distance variables) and be a finite value.

The debug checks should cover most common errors when editing the graph descriptors.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-18 11:49:30
More updates on the graph:

- Some speed improvements (below 3 secs for 40 K tracks on i7 920 from 2008).
- Added extensive documentation for its use and aim, check below.

Code: [Select]
/*	
Search by Distance v 0.2.1 17/02/21
-----------------------------------
Creates a playlist with similar tracks to the currently selected one according to genre, style, key, etc.
Every track of the library is given a score according to those tags and/or a distance according to genre/style.
When their score is over 'scoreFilter', then they are included in the final pool.
After all tracks have been evaluated and the final pool is complete, some of them are chosen to populate
the playlist. You can choose whether this final selection is done according to score, randomly chosen, etc.
All settings are configurable on the properties panel (or set in the files when called using buttons, etc.)
Check the descriptions of the properties panel to check how the variables work.

These are the weight/tags pairs checked by default:
- genreWeight : genre - styleWeight : style
- dyngenreWeight : virtual tag - moodWeight : mood
- keyWeight : key - dateWeight : $year(%date%)
- bpmWeight  : bpm - composerWeight : composer (unused by default)
There are 2 custom tags which can be set by the user too:
- customStrWeight : (unused by default) - customNumWeight : (unused by default)

Any Weight/tags pair can be remapped and/or merged (sep. by comma).
For example, linking genreWeight to 2 different genre tags on your files:
- genreTag : allmusic_genre,my_genre_tag

Some weight/tags pairs can be linked to TitleFormat Expr. Use tag names instead of TF expressions when possible (+ performance).
For example, see dateWeight: TF is used to have the same results for tracks with YYYY-MM tags or YYYY tags.
- dateTag : $year(%date%) -customNumWeight : (unused by default)

Genre and Style tags (or their remapped values) can be globally filtered. See 'genreStyleFilter'. Case sensitive.
For example, when comparing genre values from track A to track B, 'Soundtrack' and 'Radio Program' values are omitted:
- genreStyleFilter: Soundtrack,Radio Program

There are 3 methods to calc similarity: WEIGHT, GRAPH and DYNGENRE.
- WEIGHT: -> Score
Uses genreWeight, styleWeight, moodWeight, keyWeight, dateWeight, dateRange, bpmWeight, bpmRange, composerWeight, customStrWeight, customNumWeight + scoreFilter
Calculates similarity by scoring according to the tags. Similarity is calculated by simple string matching ('Rock' != 'Soul') and ranges for numeric tags. This means some coherence in tags is needed to make it work, and the script only works with high level data (tags) which should have
been added to files previously using manual or automatic methods (like MusicBrainz Picard, see note at bottom).
- GRAPH: -> Score + Distance
Same than WEIGHT + max_graph_distance
Apart from scoring, it compares the genre/styles tags set to the ones of the reference track using a graph and calculating their min. mean distance.
Minimum mean distance is done considering (A,B,D) set matches (A,B,C) at 2 points (0 distance). So we only need to find the nearest point
from (A,B,D) to (C) which will be x. Then we calculate the mean distance dividing by the number of points of the reference track : (0 + 0 + x)/3
Imagine Google maps for genre/styles, and looking for the distance from Rock to Jazz for ex. 'max_graph_distance' sets the max distance allowed, so every track with genre/styles farther than that value will not be included in the final pool. Note this is totally different to simple string matching, so 'Acid Rock' may be similar to 'Psychedelic Rock' even if they are totally different tag values (or strings).
This method is pretty computational intensive, we are drawing a map with all known genres/styles and calculating the shortest path between the reference track and all the tracks from the library (after some basic filtering). Somewhere between 2 and 5 secs for 40 K tracks.
For a complete description of how it works check: 'helpers/music_graph_descriptors_xxx.js'
And to see the map rendered in your browser like a map check: 'Draw Graph.html'
- DYNGENRE: -> Score
Same than WEIGHT + dyngenreWeight
Uses a simplification of the GRAPH method. Let's say we assign a number to every "big" cluster of points on the music graph, then we can simply
put any genre/style point into any of those clusters and give them a value. So 'Rock' is linked to 3, the same than 'Roots Rock' or 'Rock & Roll'.
It's a more complex method than WEIGHT, but less than GRAPH, which allows cross-linking between different genres breaking from string matching.
For a complete description of how it works check: 'helpers/dyngenre_map_xxx.js'

Arguments: (genreWeight, styleWeight, dyngenreWeight, moodWeight, keyWeight, dateWeight, dateRange, bpmWeight, bpmRange, playlistLength, probPick, scoreFilter, max_graph_distance, bRandomPick, method, forcedQuery)

A: do_searchby_distance(15, 10, 0, 5, 10, 25, 15, 5, 25, undefined, 100, 60, 150, 1, "GRAPH", undefined)
// Random mix with only nearest tracks, most from the same decade

B: do_searchby_distance(10, 10, 0, 5, 5, 25, 15, 5, 25, undefined, 100, 60, 250, 1, "GRAPH", undefined)
// Random mix a bit varied on styles (but similar genre), most from the same decade

C: do_searchby_distance(0, 5, 0, 15, 10, 25, 15, 5, 25, undefined, 100, 50, 600, 1, "GRAPH", undefined)
// Random mix even more varied on styles/genres, most from the same decade

D: do_searchby_distance(0, 5, 0, 15, 10, 0, undefined, 5, 25, undefined, 100, 50, 900, 1, "GRAPH", undefined)
// Random mix with different genres but same mood from any date

Note about genre/styles:
GRAPH method doesn't care whether "Rock" is a genre or a style but the scoring part does! Both values are considered points without any distinction.
Genre weight is related to genres, style weight is related to styles.... But there is a workaround, let's say you only use genre tags (and put all values
together there). Then set style weight to zero. It will just check genre tags and the graph part will work the same anyway.

Note about GRAPH/DYNGENRE exclusions:
Apart from the global filter (which applies to genre/style string matching for scoring purpose), there is another filtering done when mapping
genres/styles to the graph or their associated static values. See 'map_distance_exclusions' at 'helpers/music_graph_descriptors_xxx.js'.
It includes those genre/style tags which are not related to an specific musical genre. For ex. "Acoustic" which could be applied to any genre.
They are filtered because they have no representation on the graph, not being a real genre/style but a musical characteristic of any musical composition.
Therefore, they are useful for similarity scoring purposes but not for the graph. That's why we don't use the global filter for them.
This second filtering stage is not really needed, but it greatly speedups the calculations if you have tons of files with these tags!
In other words, any tag not included in 'helpers/music_graph_descriptors_xxx.js' as part of the graph will be omitted for distance calcs,
but you save time if you add it manually to the exclusions (otherwise the entire graph will be visited trying to find a match).

Note about GRAPH substitutions:
The graph created follows a genre/style convention found at 'helpers/music_graph_descriptors_xxx.js'. That means than only things written there, with exactly
the same name (including casing) will be found at the graph. As already noted, 'rock' will not be the same than 'Rock', neither 'Prog. Rock' and 'Progressive Rock'. This is a design decision, to force users to use a tag convention (whatever they want) and only one. See last note at bottom. As a workaround, since you
don't have to follow my convention, there is a section named 'style_substitutions' which lets you tell the scripts that 'Prog. Rock' is the same than
'Progressive Rock' for example. So once you have all your library tagged as you please, you can either mass replace the different terms to follow the convention
of the graph OR add substitutions for all of them. It has other complex uses too, but that goes beyond this doc. Check 'helpers/music_graph_descriptors_xxx.js'.

Note about console logs:
The function assigns a score to every track and you can see the names - score (- graph distance) at console. The selected track will get a score = 100 and distance = 0. Then, the other tracks will get (lower or eq.) score and (greater or eq.) distance according to their similarity. You can toggle logging and profiling on/off setting the booleans within the function code (at their start).

Note about graph configuration and error checking:
By default 'bGraphDebug' is set to true. That means it checks the graph files whenever you load the script (usually once per session). You can disable it (faster loading). If you edit 'helpers/music_graph_descriptors_xxx.js' file, it's advised to enable it at least once so you can check there are no obvious
errors. The html rendering helps at that too (and also has debugging enabled by default).

Note about buttons framework:
When used along buttons framework, you must pass all arguments, along the new prefix and merged properties! See "buttons_search_bydistance_customizable.js"

Note about High Level data, tag coherence, automatic tagging and MusicBrainz Picard:
Network players and servers like Spotify, Itunes Genius, YouTube, etc. offer similar services to these scripts. Whenever you play a track within their players,
similar tracks are offered on a regular basis. All the work is done on the servers, so it seems to be magic for the user. There are 2 important caveats for
this approach: It only works because the tracks have been previously analyzed ('tagged') on their server. And all the data is closed source and network
dependent. i.e. you can always listen to Spotify and your great playlist, at least while you pay them, those tracks are not removed for your region and you
have a constant Internet connection.

That music listening model has clear drawbacks, and while the purpose of this caveat is not talking about them, at least we have to note the closed source
nature of that analysis data. Spotify's data is not the same than Youtube's data... and for sure, you can not make use of that data for your library in any way.
Ironically, being at a point where machine learning and other methods of analysis are ubiquitous, they are mostly relegated behind a pay-wall. And every time
a company or a developer wants to create similar features, they must start from scratch and create their own data models.

An offline similar program which does the same would be MusicIP ( https://spicefly.com/article.php?page=what-is-musicip ). It appeared as a viable alternative
to that model, offering both an Internet server and a complete ability to analyze files offline as fall-back. Nowadays, the company is gone, the software
is obsolete (although usable!) and documentation is missing for advanced features. The main problems? It was meant as a standalone solution, so there is no
easy way to connect other programs to its local server to create playlists on demand. It can be done, but requires manually refreshing and maintaining
the server database with new tag changes, data analysis, and translating ratings (from foobar for ex.) to the program. The other big problem is analysis time.
It may well take a minute per track when calculating all the data needed... and the data is also closed source (so it has no meaning outside the program).
The reason it takes so much time is simple, the program was created when machine learning was not a reality. MusicIP may well have been ahead of its time.

Back to topic, both online and offline methods due to its closed source nature greatly difficult interoperability between different programs and use-cases.
These scripts offer a solution to both problems, relying only in offline data (your tags) and open source data (your tags again). But to make it work,
the data (your tags) need relevant info. Since every user fills their tags without following an universal convention (most times leaving some tags unfilled),
the only requirement for these scripts to work is tag coherence:
- Tags must point to high-level data, so analysis (whether computational or human) must be done previously. 'Acoustic' or 'Happy' moods are high level data,
'barkbands' with a list of values is low-level data (meant to be used to calculate high-level ones). See this for more info: https://acousticbrainz.org/data
- Tags not present are simply skipped. The script doesn't expect any specific tag to work, except the obvious ones (can not use GRAPH method without
genre/style tags). This is done to not enforce an specific set of tags, only the ones you need / use.
- Tags are not hard-linked to an specific tag-name convention. Genre may have as tag name 'genre', 'my_genre' or 'hjk'.
- Basic filtering features to solve some corner cases (for genre/styles).
- Casing must be the same along all tags. i.e. 'Rock' always spelled as 'Rock'. Yes, it could be omitted using .toLowerCase(), but is a design decision, since it forces users to check the casing of their entire set of tags (ensuring a bit more consistency in other fields).
- Reproducibility all along the library set. And this is the main point of tag coherence.

About Reproducibility: If two users tag a track with different genres or moods, then the results will be totally different. But that's not a problem as long as
every user applies its own 'logic' to their entire library. i.e. if you have half of your library tagged right and the other half with missing tags,
some wrongly set and others 'as is' when you got the files... then there is no coherence at all in your set of tracks nor your tags. Some follow a convention
and others follow another convention. To help with that here are 2 advises: tag your tracks properly (I don't care about specific use-cases to solve what ifs)
and take a look at MusicBrainz Picard (that's the open source part): https://picard.musicbrainz.org/
Now, you DON'T need to change all your tags or your entire library. But Picard offers 3 tags using the high-level open source data of AcousticBraiz:
mood, key and BPM. That means, you can use your manual set tags along those automatically set Picard's tags to fulfill both: your tag convention and reproducibility along your entire library. Also you can manually fix or change later any mood with your preferred editor or player. Picard also offers plugins
to fill other tags like genres, composers, etc. Filling only empty tags, or adding them to the existing ones, replacing them, ... whatever you want.
There are probably other solutions like fetching data from AllMusic (moods), lastFm (genres), Discogs (composers), etc. Use whatever you want as long as tag
coherence and reproducibility are ensured.

What happens if you don't want to (re)tag your files with moods, bpm, key, splitting genres/styles, ...? Then you will miss that part, that's all. But it works.
What about a library properly tagged but using 'rock' instead of 'Rock' or 'African Folk' instead of 'Nubian Folk' (although that's a bit racist you know ;)? Then use substitutions at 'helpers/music_graph_descriptors_xxx.js' (not touching your files) and/or use mass tagging utilities to replace values (touching them).
And what about having some tracks of my library properly tagged and not others? Then... garbage in, garbage out. Your mileage may vary.
*/
/*
  Last changes:
- Speed improvements replacing arrays with sets in multiple places.
- Speed improvements on tag filtering.

TODO:
- Pass object as argument
- Overwrite substitutions with user set file:
- Put this after main file
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\helpers\\music_graph_descriptors_xxx_user.js');
Merge old object property with the ones set there.
On new updates, only overwrite main file.
- Clean unused things
- Fingerprint comparison for scoring?
- Output just similar tracks by fingerprint
- Give more weighting to similar tracks
- Fine-tune Pre-filtering:
- Fine-tune queries
- GRAPH : Apart from individual link caching, cache entire distance from set (A,B,C) to (X,Y,Z)
- Different date formats or just TF? YYYY-MM-DD; YYYY-MM
- Make it easy to port the code to other frameworks outside foobar:
- Include are just replaced with import
- Properties are just objects
- All graph calcs and descriptors are independent of foobar so they work 'as is'. That's the main feature and heaviest part.
- All tag 'getValues' would be done using external libraries or native player's methods
- Queries would be disabled (they are not intrinsic part of the code, just a speed optimization)
- Handle lists should be replaced with similar objects
- Console logs work as they are.
- Helpers used don't need foobar at all (except the 'getValues' for tags obviously)
*/

Next I will add a short FAQ and how-to for adding new releases into AcousticBrainz, so anyone can get all data (mood,key, bpm) for their own files even if they were not in the server. Following a simple routine and using a Picard Plugin , it takes 3 minutes of user intervention.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-19 12:59:58
More updates on the graph:
- Added user file to overwrite\add\delete properties from 'music_graph_descriptors'. On new updates, it would never get overwritten... making it easy to maintain your own changes without having to manually merge files (or breaking the main file). If the value is added, deleted or replaced is done according to the logic described at 'music_graph_descriptors_xxx_user.js'.

Added this to the doc:
Code: [Select]
	Note about editing 'helpers/music_graph_descriptors_xxx.js' or user file:
Instead of editing the main file, you can add any edit to an user set file named 'helpers/music_graph_descriptors_xxx_user.js'. Check sample for more info.
It's irrelevant whether you add your changes to the original file or the user's one but note on future script updates the main file may be updated too.
That means you will need to manually merge the changes from the update with your own ones, if you want them. That's the only "problem" editing the main one.
Both the html and foobar scripts will use any setting on the user file (as if it were in the main file), so there is no other difference.
Anything at this doc which points to 'helpers/music_graph_descriptors_xxx.js' applies the same to 'helpers/music_graph_descriptors_xxx_user.js'.

Then a bit more info in the sample file:
Code: [Select]
/*
This is a sample file to show how to replace/add/delete properties at music_graph_descriptors without touching the original file.
Just copy properties from the original file 'helpers\music_graph_descriptors_xxx.js' to this one and edit whatever you want.
The replacement follow this logic:
-Always replaced: Strings , numbers, booleans
-Adds new values: new elements on arrays and sets.
-Replaces old values with new ones: elements present on both arrays
-Deletes old values: elements present on both sets
Note properties need a ',' after the values:
propertyA: value,
propertyB: 'valueB',
propertyC: [valueC, [subValuesC]],
...
*/

const music_graph_descriptors_user = {
/*
// You can either replace, delete or add array values to the original one
// This example would replace the original value with a new one.
style_supergenre: [
['Classical Modernist Era_supergenre', ['Third Stream']]
],

// This example would simply delete the old value (= replacing it with empty values)
style_supergenre: [
['Classical Modernist Era_supergenre', []]
],

// This example would add a new value
style_supergenre: [
['My Own_supergenre', ['My Own Style/Genre A', 'My Own Style/Genre B']]
],
*/

/*
// You can either delete or add set values to the original one
// Deletes 'Female Vocal' (since it was already present) and adds 'My New Set Value'
map_distance_exclusions: new Set([
'Female Vocal','My New Set Value'
]),
*/

/*
// You can replace numbers
primary_origin: 200,

// strings
nodeShape: 'circle',

// and boolean properties
bPreRender: false,
*/
};

The main point of this change is not only adding your own values... but also replacing or deleting default ones without touching the original graph descriptors. And doing it without complex edits, just doing the same than adding values.

It works both on html rendering and within foobar, so it's really easy to see if they work as intended just by refreshing the html file (for example adding new genres, changing distances, etc.). Since I added extensive debug functions on the previous updates, the user set values are also checked for consistency.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-23 11:38:57
More updates on the graph:
- Added extensive examples to use the script for different playlist mixes.
- Default arguments using object notation (easier to set only those you want).
- Pre-scoring filters
   - Anti-Influence filter: filter anti-influences by query for GRAPH Method.
   - Influence filter: gets only influences by query for GRAPH Method.
- Post-scoring filters
   - Allow only N+1 tracks per tag on the pool. Configurable. i.e. 2 tracks per artist, 4 tracks per genre, etc.
- Link cache pre-calculated on startup with styles/genres present on library (instead of doing it on function evaluation later).

Code: [Select]
/*	Arguments: 	As object, anything not set uses default values set at properties panel.
{ properties,
// --->Weights         
genreWeight, styleWeight, dyngenreWeight, moodWeight, keyWeight, dateWeight, bpmWeight, composerWeight, customStrWeight, customNumWeight,
// --->Ranges (for associated weighting)
dyngenreRange, dateRange, bpmRange, customNumRange,
// --->Pre-Scoring Filters
forcedQuery, // Query to filter library
bUseAntiInfluencesFilter, // Filter anti-influences by query, before any scoring/distance calc. Stricter than distance checking.
bUseInfluencesFilter, // Allow only influences by query, before any scoring/distance calc.
// --->Scoring Method
method, // GRAPH, WEIGHT, DYNGENRE
// --->Scoring filters
scoreFilter, sbd_max_graph_distance, // Minimum score & max distance to include track on pool
// --->Post-Scoring Filters
poolFilteringTag, poolFilteringN, bPoolFiltering, // Allows only N +1 tracks per tag set on pool... like only 2 tracks per artist
// --->Playlist selection
bRandomPick, // Get randomly from pool
probPick, // Get by scoring order but with x probability of being chosen from pool
playlistLength,
bSortRandom, // Final sorting (independently of picking method)
// --->Console
bProfile, bShowQuery, bShowFinalSelection // Different logs on console
}

Examples: Some usage examples, most of them can be combined in some way (like A with H, etc.)
A:  Random mix with only nearest tracks, most from the same decade
args = {genreWeight: 15, styleWeight: 10, moodWeight: 5, keyWeight: 10, dateWeight: 10, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, sbd_max_graph_distance: 150, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

B: Random mix a bit varied on styles (but similar genres), most from the same decade. [like A with more diversity]
args = {genreWeight: 10, styleWeight: 5, moodWeight: 5, keyWeight: 5, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, sbd_max_graph_distance: 250, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

C: Random mix even more varied on styles/genres, most from the same decade.
args = {genreWeight: 0, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 50, sbd_max_graph_distance: 300, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

D: Random mix with different genres but same mood from any date.
args = {genreWeight: 0, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 0, bpmWeight: 5, bpmRange: 25,
probPick: 100, scoreFilter: 50, sbd_max_graph_distance: 600, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

E:  Uses the properties of the current panel to set all the arguments.
do_searchby_distance();

F:  Uses a properties object to set all the arguments. [like E but overriding the properties used, usually for merged buttons]
args = {properties: yourPropertiesPairs}; // {key: [description, value], ...} see SearchByDistance_properties
do_searchby_distance(args);

G:  Outputs only influences from any date. [doesn't output similar genre/style tracks but influences! May be similar too, but not required]
args = {genreWeight: 5, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 0, bpmWeight: 10, bUseInfluencesFilter: true,
probPick: 100, scoreFilter: 40, sbd_max_graph_distance: 500, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

H:  Uses the properties of the current panel, and allows only 2 tracks (n+1) per artist on the pool. [like E with pool filtering]
args = {poolFilteringTag: 'artist', poolFilteringN: 1} // bPoolFiltering evaluates to true whenever poolFilteringN != -1
do_searchby_distance(args);

I:  Random mix even more varied on styles/genres, most from the same decade. [like C but WEIGHT method]
args = {genreWeight: 10, styleWeight: 5, moodWeight: 5, keyWeight: 5, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, bRandomPick: true, method: 'WEIGHT'};
do_searchby_distance(args);

J:  Mix with similar genre/styles using DYNGENRE method. The rest of the args are set according to the properties of the current panel.
args = {dyngenreWeight: 20, dyngenreRange: 1, method: 'DYNGENRE'};
do_searchby_distance();*/

Code: [Select]
/*	TODO:
- Replicate progressive lists and intelligent list ordering, saving moods and genre/styles not in common.
-Then selecting tracks for final playlist according to those values; like 4 tracks with set (A,B,C), then change 1 variable, then another 4 tracks, etc.
-Not grouping instrumental tracks if possible, etc.
- Clean unused things
- Fingerprint comparison for scoring?
- Output just similar tracks by fingerprint
- Give more weighting to similar tracks
- Fine-tune Pre-filtering:
- Fine-tune queries
- GRAPH : Apart from individual link caching, cache entire distance from set (A,B,C) to (X,Y,Z)
- Cache all links below a threshold? Instead of caching all possible links.
- Different date formats or just TF? YYYY-MM-DD; YYYY-MM
- Get data using MusicBrainz Ids for all tracks on library, save to json files and use them when no tags present
- Support for more high-level data (https://acousticbrainz.org/data#sample-data):
- danceability
- gender
- moods_mirex
- timbre
- tonal_atonal
- voice_instrumental
- Instruments
- Make it easy to port the code to other frameworks outside foobar:
- Include are just replaced with import
- Properties are just objects
- All graph calcs and descriptors are independent of foobar so they work 'as is'. That's the main feature and heaviest part.
- All tag 'getValues' would be done using external libraries or native player's methods
- Queries would be disabled (they are not intrinsic part of the code, just a speed optimization)
- Handle lists should be replaced with similar objects
- Console logs work as they are.
- Helpers used don't need foobar at all (except the 'getValues' for tags obviously)*/
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-24 10:24:28
Continuing with the Graph Scripts, I'm now working on MusicIP functionalities:
-Recipes should be fully covered with current weigh/tag/queries options, since you can check any tag, set of tags, TF expressions, etc. And also pre/post filter the pool. (I would say the script gives even more possibilities)
https://www.spicefly.com/article.php?page=musicip-recipes

-Moods are trivial to replicate using a set of songs as reference. The missing 'part' would be storing those reference internally (json) instead of having the tracks on a playlist. Also I'm using only 1 track as reference, but it should be easy to 'merge' the output from a set of tracks, and only send to final playlist those which are more similar to all selected references.
https://www.spicefly.com/article.php?page=musicip-moods

-Right now I want to integrate some of the advanced playlist creation features. The current graph scoring method is more advanced than MusicIp similarity method, but it could improve on other aspects (UI, pre-defined options, etc.).

I would appreciate some ideas or comments on the last points, now that I consider adding them.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-25 21:11:24
Last changes implemented: added the 3 features I mentioned about MusicIp.

   - Scattering vocal & instrumental tracks, breaking clusters of instrumental tracks.
   - Progressive list ordering, having more different tracks the more you advance within a playlist (using probPick < 100 && bProgressiveListOrder). Can be combined with any weight/filter preference, to suit anyone's need.
   - Progressive playlist creation, uses one track as reference, and then uses the output tracks as new references, and so on... takes more time than standard search, since it performs recursive calls...

The last feature is working pretty great, using an acoustic stoner rock song, creates a playlist which starts with stoner/hard rock tracks and then progressively changes to folk-rock tracks and psychedelic. And since it can be merged with any of the other options, you can pretty much create smart playlists in any imaginable way. With blues, it goes from blues to jazz, jazz vocal, blues, etc. Only downside is calc time, but it can be made shorter using WEIGHT method instead of GRAPH, which takes less time. Also using queries to prefilter library.
Title: Re: [SMP] Music Graph development
Post by: 2tec on 2021-02-25 22:42:30
Pardon me for asking but do you have plans for a public beta?
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-26 00:06:09
Of course, as as soon as I see it's stable enough :) (which means, not susceptible to fundamental changes) I'm changing files every day, so it makes no sense to me publishing them right now publicly... that's all (since I may break how things worked 3 days ago.) I have time to code, but not for support and/or explaining every change right now. It's not like I can upload them to github and done, since I would have to change the documentation every week.

But drop me a line and I will give you (or anyone) the current version.
regorxxx@protonmail.com
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-02-28 21:42:44
Someone asked me about the key notation I used for similarity matching and at some point Camelot Wheel was mentioned, I have been working on it and have translated that logic (the rule of fifths) into scripts.
https://pyramind.com/harmonic-mixing-using-camelot-system-mixed-in-key/

DJs use it to create mixes/playlists with tracks which are considered 'harmonically compatible'. Usually traveling the wheel following horizontal or vertical lines in +1 steps. (there are other complex movements too). Have added that to the GRAPH script, now when using a track as reference, the key is converted to camelot system notation and it checks for all tracks with keys within that cross in a range (set at properties). Therefore creating playlists the same a DJ would do.
Obviously, that is just the key weighting... so it can be used standalone or along genre, style, dyngenre weightings,  graph method, etc.

      - Keys are supported using standard notation (Ab, Am, A#, etc.) and flat or sharp equivalences (both are translated to the same values). This standard is according to the tags acousticBrainz, piccard gives you... and the one used universally  inmusic.
      - Key matching is done using camelot wheel logic, allowing similar keys by a range using a 'cross' (changing hour or letter, but both is penalized).
      - For other key notations simple string matching will be used. That means I don't support abstract notations, like Camelot or open keys for the 'wheel logic'.(*)

(*) The reason is simple, I have no way to know if 1A, 3B, etc. is following Camelot or Open Keys notation, and since there are already scripts to convert tags to one or other system... that's user responsibility and I just use universal notation. Anyway dynamic tags can be used to create a converted key tag and I may remap the tag to add TF support too.

TODO:
      - Add key logic to 'search_similar_by', to create compatible queries instead of similarity checks. (a simplified version of the graph scripts with just queries)
      - Use current progressive list logics to create intelligent playlists with incremental keys, complex movements, etc.

Below is an example of how to set the properties to check for all tracks on library with similar genre/styles, using simple weight method and only allow those with similar Keys (on cross with 1-range).
X

X

As shown on the properties panel, the keys of the tracks in the output are: Am, C, Em. Right considering I used a track with C key as reference.
(https://pyramind.com/wp-content/uploads/2020/05/preview-lightbox-CamelotWheel-1.jpg)


Quote
-Moods are trivial to replicate using a set of songs as reference. The missing 'part' would be storing those reference internally (json) instead of having the tracks on a playlist. Also I'm using only 1 track as reference, but it should be easy to 'merge' the output from a set of tracks, and only send to final playlist those which are more similar to all selected references.
https://www.spicefly.com/article.php?page=musicip-moods
And this is postponed until foobar native playlist format becomes public or SMP (@TheQwertiest ?) improves foobar playlist saving/loading via native methods... since I can not save a group of tracks as a mood and then load it without user intervention or loading the playlist on UI to get the handle. And it would finally make the playlist manager fully functional, supporting the native foobar playlist.

PD: I don't answer PMs here, I have given an email for a reason ;)
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-04 09:15:28
New update:
-  New Option: DJ-like playlist creation with key changes following harmonic mixing rules. The entire pool (according to configurable weights, etc.) is considered, instead of using the standard playlist selection (randomly, by similarity score, etc.).

An explanation of how harmonic mixing works can be found here:
https://www.bestdjgear.net/harmonic-mixing-rules-everything-you-need-to-know/

An example would be having a track with key 7B as reference then the next 5 tracks on the playlist would have keys:
8B, 8A, 9A, 9B, 8B
X
And that would continue until it reaches the desired playlist length.

Harmonic mixing is usually done following 9 possible 'key movements', so I have added methods to describe things like  'Energy Boost' which is equivalent to moving up 1 hour in the Camelot Wheel, etc. Everytime you want a new playlist, a new pattern is created. Instead of presetting a series of movements, they are chosen randomly according to a given proportion. The result is a new playlist not only with different tracks but with its own structure everytime (so you don't expect a mood change, or a boost always at X track).

For people who like electronic music, this could replace having someone choosing the next tracks, as long as you choose an electronic track as reference (or using 'forced query ' option to only consider that genre). Anyway,it works for any genre, creating natural progressions between tracks.


So let's resume the current implementation of the Weight/graph script:
- Creates a playlist with similar tracks to the currently selected one according to genre, style, key, etc or following special rules.
- The tags to check are associated to a weight, currently they are: genre,  style, dynGenre, mood,  key,  $year(%date%), bpm, composer, customStringTag, customNumTag
- Tags can be remapped: genre ->allmusic_genre, my_genre_tag
- There are 3 methods to calculate similarity: WEIGHT, GRAPH and DYNGENRE.
- WEIGHT and DYNGENRE assign scores based on tag comparison, checking numbers within a range or complex logic comparison (i.e. keys using camelot wheel).
- GRAPH assigns genres/styles a place in a graph based on their relations, and calculates similarity by the shortest path which links them. Classifying all known musical genres is required, pretty computational expensive and is an ongoing work. (there are multiple images on the thread)
- Tracks are added to a pool if they pass the score/distance check. (i.e. score >= 70%)
- There are complex settings to filter, force queries, remove duplicates or allowing only X tracks per tag on the pool.
- Finally, tracks are selected from the pool to create a playlist. This can be done randomly, following strict scoring order, or starting with highest scored tracks but with a probability of being chosen.
- Apart from that, there are 2 more special playlist creation rules: Harmonic Mixing (previous explanation) and Progressive List Creation (recursive calls to playlist creation using new similar tracks as references) .
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-03-14 21:19:08
Hi! regor you sent me some scripts in February, in 15 February to be more precisely. Yes, I like Top 25 Tracks , it's fast (actually I modified the script a bit, now it's Top 100). In the Playlist Manager - Add new empty playlist file , I have this error :
Playlist generation failed while writing file 'D:\foobar2000\playlist_manager\playlist.m3u8'
Except this error the Playlist Manager works fine, but as I said , I can't create new playlists. I don't know what might be, I have the last version of foobar portable and SMP.
I assume the scripts are now a bit different as you worked on them , so you can send me the scripts if they're different.

At last but not the least, we talked about a script for Most played song in a year (2017, 2015, or any other year) https://hydrogenaud.io/index.php?topic=115227.msg994727#msg994727
Can you do that script?
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-14 22:24:10
Hi! regor you sent me some scripts in February, in 15 February to be more precisely. Yes, I like Top 25 Tracks , it's fast (actually I modified the script a bit, now it's Top 100).
You don't need to modify the script for that, at least if you use it within a button, Just change the associated property at the panel:
X
X
And if you are using the script function via menu or other scripts, then just change the argument.
Code: [Select]
do_top_tracks(100) 

No need touch the files at all, unless you have done other changes. Sorting and duplicates deletion are arguments too. Defaults:
Code: [Select]
do_top_tracks( number = 25, 	sortBy = "$sub(99999,%play_count%)",  checkDuplicatesBy = ["title", "artist", "date"]) 
Quote
In the Playlist Manager - Add new empty playlist file , I have this error :
Playlist generation failed while writing file 'D:\foobar2000\playlist_manager\playlist.m3u8'
Except this error the Playlist Manager works fine, but as I said , I can't create new playlists. I don't know what might be, I have the last version of foobar portable and SMP.
Not enough details... for sure I can create a new playlist without problems.
Does 'D:\foobar2000\playlist_manager\' exist?

That error only fires if the script fails to create that file 'playlist.m3u8'. Even before saving data to the file. Check the folder since that seems to be the problem. I have not added any logic to create the folder, the user must check the desired folder exists ;) (*)
 (my reasoning was the User should always configure the panel to point to their own playlist folder path, so pointing to a non existent path -like the default one- made no sense. If you wanted to use the default one, then it had to be created too.)
 
Quote
At last but not the least, we talked about a script for Most played song in a year (2017, 2015, or any other year) https://hydrogenaud.io/index.php?topic=115227.msg994727#msg994727
Can you do that script?
Will work on it this week.

(*) Anyway just added that check, so the first time a playlist is created if the folder does not exist, it's created too which is more intuitive. Thanks for the report!
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-03-15 15:49:06
 Yes, I changed the associated property at the panel (Top 25 Tracks) , but I said I modified the script, sorry. (You are more technically, for me it's almost the same thing, haha :) Sorry once again, it's my mistake.)

With the Playlist Manager, you were right, the folder playlist_manager wasn't there, I didn't create the folder. But I'm glad you solved this little problem.
About the script for Most played song in a year....What can I say...Yesss ! :) Thanksss
I can't wait , I think it's a interesting feature (script) that it's not present in other audio players (MusicBee, jriver or aimp)

Another little problem about the Playlist Manager :
"Check playlist loaded, there are duplicated names. You can not have duplicates if using autosave. Names:
Trance, Untitled, New music"
These duplicate playlists (Trance, Untitled, New music) are not in your Playlist Manager, they're in Foobar, Playlist Switcher.
This error appears when I drag and drop a few files in one of those duplicate playlists. I don't know if it's such a big problem, I could rename those playlists but I thought you should know about this.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-15 15:56:21
Yes, I changed the associated property at the panel (Top 25 Tracks) , but I said I modified the script, sorry. (You are more technically, for me it's almost the same thing, haha :) Sorry once again, it's my mistake.)

With the Playlist Manager, you were right, the folder playlist_manager wasn't there, I didn't create the folder. But I'm glad you solved this little problem.
About the script for Most played song in a year....What can I say...Yesss ! :) Thanksss
I can't wait , I think it's a interesting feature (script) that it's not present in other audio players (MusicBee, jriver or aimp)

Another little problem about the Playlist Manager :
"Check playlist loaded, there are duplicated names. You can not have duplicates if using autosave. Names:
Trance, Untitled, New music"
These duplicate playlists (Trance, Untitled, New music) are not in your Playlist Manager, they're in Foobar, Playlist Switcher.
This error appears when I drag and drop a few files in one of those duplicate playlists. I don't know if it's such a big problem, I could rename those playlists but I thought you should know about this.
There was a similar error some time ago, but I don't know if I fixed it after sharing the files or later.

When adding tracks to playlists, ONLY if you have autosave enabled, the manager checks every playlist loaded in foobar to find a match in the playlist manager. That's why auto-saving is incompatible with duplicates names.

To solve that problem (associating an unique ID to every playlist), you must use different names or use the UUID features of the panel (right click).

Now, if you have playlist with totally different names loaded, then there should be no problems. Please, check you don't have duplicates on the manager too (check the files on the playlist folder, and their internal names with any text editor). If everything is right, then it's probably that bug I mentioned (and I will give you the latest version to test if it's solved)

EDIT: Reading you again, I think you are talking about real duplicates. Right?  i.e. you have duplicates playlist on foobar, but you don't have them on the manager at all. Yep. I check for duplicates on both sides! I could "allow" duplicates for playlist not present on the manager, although I would add it as an option and not the default behavior when using the manager (because it would be weird to have a mix of unique and duplicates playlists for regular users) .
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-03-15 16:07:33
Yes, I have duplicate playlists on foobar, but I don't have them on the manager at all
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-15 16:28:28
Then it's not a "bug". Autosave feature forces you to not have duplicates at all. You can either turn off autosaving or rename the duplicated playlists. Anyway just added the "fix", a few lines of code, thanks! (I will decide later if it becomes the default behavior or just an option)
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-15 17:41:03
Quote
About the script for Most played song in a year....What can I say...Yesss ! :) Thanksss
I can't wait , I think it's a interesting feature (script) that it's not present in other audio players (MusicBee, jriver or aimp)
Got it working easily BUT there are 2 problems

- %played_times% and %lastfm_played_times% are independent variables, should the script check both? For now, I have merged them.


- The main problem: enhanced playcount only adds statistics to those variables after being installed AND tracks being played at least once. That means the script provides strange playlists when some tracks are missing those statistics. What should happen when %played_times% is empty? Should I use the standard %play_Count% or skip the track? For now, I have simply put the tracks with %played_times%. The rest is skipped.

(*) Imagine you have played track A 100 times last year, but %played_times% is empty because you just installed the component today. Using only %played_times% you would not get track A on Top N tracks playlist... using standard queries (playCount), it would be there.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-15 21:01:05
Got it. It requires the helpers from the other scripts and the buttons framework (for the button), so you must add these new files to the others. It should overwrite 2 files top_tracks.js and buttons_search_top_tracks.js
Sorry for not giving all dependencies and a standalone release, but I alreay warned this was a WIP yet. Will do asap.

As a bonus, I added a pre-filter query to both the top tracks from X year, and the standard top Tracks (from any year) scripts.
Also updated the buttons to show the pre-filter info, which can be set at the properties panel.
The playlist  for top tracks from X year script uses the year as name too.
And the year is automatically set (on the button) using current year - 1.

X

The script follows this logic:
- pre-filter query
- %last_played% > year AND %first_played% < year + 1
- Sort by total play count
- Counts %played_times% and %lastfm_played_times% within the year and merges both.
- If those arrays are null, because the user has no enhanced component or statistics updated for the tracks then:
      - use all play counts IF both %last_played% AND %first_played% == year
      - else count 1 IF %last_played% == year
      - else count 1 IF %first_played%  == year

Found the last part the best solution for those not having enhanced statistics, since it retrieves true play count if you did not played the track later than the selected year, and at least adds a play count if first/last play equals that year.

Also tracks are first sorted by play count, so even for those not having the enhanced statistics component, the script should give the best approximation to the top tracks from that year (assume the most played tracks are also the most played tracks from any year, if they were played at least once that year).
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-16 14:24:35
Well, using the package manager of SMP I have created a temp standalone release for the previous script (top X tracks from year). No need for other files. Check attachment. Requires latest version of SMP. For previous versions, you need to use the standard js files (previous post).

X

X
Title: Re: [SMP] Music Graph development
Post by: MordredKLB on 2021-03-16 15:57:49
- %played_times% and %lastfm_played_times% are independent variables, should the script check both? For now, I have merged them.
I'm not able to check what you've done at the moment, but the component strives to ensure that the recorded times from %played_times% and %lastfm_played_times% are the same (I subtract 60 seconds from the l.fm scrobble times before saving them).

Best thing to do would probably be to get %played_times_js% and %lastfm_played_times_js%, concat the two arrays, sort, and then remove duplicates (or even duplicates within 60000ms of each other).
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-03-16 16:43:13
Sorry for late replay, I wasn't at my pc. I tested the first release, everything seems to be ok (but I'm not 100% sure).I definitely see my most played songs in a particular year (let's say 2019 or other year)
It's nice to have a view for every year, to see what songs you listened the most in a year, not just overall. So, congrats!! :) Thanks!  I'm sure others will find this script useful too.
The only little problem that I have is that in this latest release I can't modify the year. Yes I have Top 25 Tracks 2020, but how can I modify to have Top 25 Tracks 2018 or other year?
I managed to modify the year in the first release but in the latest release no. Sorry if it's a dumb question
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-16 16:57:52
- %played_times% and %lastfm_played_times% are independent variables, should the script check both? For now, I have merged them.
I'm not able to check what you've done at the moment, but the component strives to ensure that the recorded times from %played_times% and %lastfm_played_times% are the same (I subtract 60 seconds from the l.fm scrobble times before saving them).

Best thing to do would probably be to get %played_times_js% and %lastfm_played_times_js%, concat the two arrays, sort, and then remove duplicates (or even duplicates within 60000ms of each other).
That's what I did.
My "problem" was %lastfm_played_times_js% was empty but %played_times_js% was not (i don't use last fm!). So my reasoning was... that last fm variable only had last fm's plays and the other variable had foobar's plays. That's why I merged them as is  (without checking duplicates).
If that's not right according to your info, then there is a bug somewhere. (or things work different for people without last fm component?) For sure, plays within foobar do not update the last fm variable too:

X
I can't check if the opposite works, since I don't use last fm.

Anyway merging them is just a matter of using sets, to remove duplicates without sorting or comparing at all. Like:
merged = [...new set(arrayA.concat(arrayB))]

Quote
The only little problem that I have is that in this latest release I can't modify the year. Yes I have Top 25 Tracks 2020, but how can I modify to have Top 25 Tracks 2018 or other year?
They are pretty equivalent, but you have to edit the files within the package (main.js) on the second release. Otherwise they are 100% the same files.

That part controls the year. It's currently set to current year - 1, but you can edit it to whatever you want. Anyway it's just a proof of concept. If it works right, my idea is to simply put the year as a property easily configurable ;) (and setting the property to nothing would use current year -1)

X
X

EDIT: note top_tracks_from_date.js has the defaults arguments, but the button file override them. You can create your own buttons, with different dates, etc. check the examples at button. The "button file" is buttons_search_top_tracks_from_date.js in the regular release, and main.js in the package release.


Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-03-16 17:22:27
Yes, it works now, thanks for explaining, I was looking in top_tracks_from_date.js and not in main.js
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-16 17:23:50
The current graph script allows for infinite possibilities of playlist creation: DJ like playlists, progressive playlists, similar by genres, by keys, by moods, by composer, etc.
Then we also have the query approach (i created 2 or 3 scripts for that). And now this Top X tracks from year.

Currently, all of them work with SMP custom menus or using buttons. I created the buttons framework to easily add/merge buttons into bars but my future aim is to create a menu panel like this:
X

Which would combine all playlist scripts within a single panel, not needing buttons for every one of them. At that point it should be pretty simple to merge anything I create with any theme. Since my idea is to augment foobar's playlist capabilities, not messing with the UI. Right now I'm just focusing on all the individual scripts for playlist creation, and later I will merge them.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-18 10:34:18
Following previous topic, I have created a contextual menu creator helper to easily add contextual menus to any panel.
Right now the usual method is creating this function
Code: [Select]
this.rbtn_up = (x, y) => {...}
And then adding manually menu entries with indexes. The main problem is later edits always require careful index checking, leaving holes for submenus, etc.
Also, every panel requires the same code. And multiple objects within the same panel require nested functions calls to append multiple menus.
A switch block is needed later to check the idx selected, so you must manually link the menu creation and the idx checking, losing even more time when coding it.
And lets not talk about changing the order of some entries at a menu once it has been already coded...



As a solution here is offered an alternative, where you simply add entries to a list and the menu is created automatically with their indexes calculated and linked to the right entries.

Code: [Select]
'use strict';

/*
Contextual Menu helper v 1.0 18/03/21
Helper to create contextual menus on demand on panels without needing to create specific methods for
every script, calculate IDs, etc. Menus are pushed to a list and created automatically, linking the entries
to their idx without needing a 'switch' block or leaving holes to ensure idx get enough numbers to expand the script.
The main utility of this helper is greatly reducing coding for simple menus and having both, the menu logic creation
and the menus' functions on the same place. Creation order is done following entry/menus addition.

Methods:
_menu({bSupressDefaultMenu = true, idxInitial = 0})
-bSupressDefaultMenu: Suppress the default context menu. left shift + left windows key will bypass it.
-idxInitial: Specifies an initial idx to create menus (useful to concatenate multiple menus objects)

.btn_up(x, y, object)
-NOTE: Called within callbacks to create the menu. Specifying an object (like another menu instance), lets you
concatenate multiple menus. Uses object.btn_up() and object.btn_up_done()
.getMainMenuName()
-NOTE: Used to get the key of the main menu. Useful to concatenate multiple menus.

.newMenu(menuName)
-menuName: Specifies the menu name or submenus names.
-NOTE: Menu is called 'main' when it's called without an argument.
-NOTE: Every menu created after the first one (main menu) will be appended to the main menu.

.newEntry({entryText = null, func = null, menuName = menuArr[0], flags = MF_STRING})
-entryText: new menu entry text. Using 'sep' or 'separator' adds a dummy separator.
-func: function associated to that entry
-menuName: to which menu/submenu the entry is associated. Uses main menu when not specified
-flags: flags for the text
-NOTE: All arguments (but 'func') may be a variable or a function (evaluated when creating the menu)

.newCheckMenu(menuName, entryTextA, entryTextB, idxFunc)
-menuName: to which menu/submenu the check is associated
-entryTextA:From entry A (idx gets calculated automatically)
-entryTextB:To entry B (idx gets calculated automatically)
-idxFunc: Logic to calculate the offset. i.e. EntryA and EntryB differ by 5 options, idxFunc must return values between 0 and 5.
-NOTE: All arguments (but 'idxFunc') may be a variable or a function (evaluated when creating the menu)

For example:
-Standard menu:
var menu = new _menu();
menu.newEntry({entryText: 'Hola', func: () => {console.log('hola')}});
menu.newEntry({entryText: 'sep'});
menu.newEntry({entryText: 'Hola2', func: () => {console.log('hola2')}});
var bSubMenu = true;
const funct = () => {return (bSubMenu) ? 'SubMenu 1' : 'SubMenu 2';};
menu.newMenu(funct);
menu.newEntry({menuName: funct, entryText: 'Change SubMenu', func: () => {bSubMenu = !bSubMenu}});
menu.newEntry({menuName: funct, entryText:'Hola 3', func: () => {console.log('hola3')}, flags: () => {return (bSubMenu) ? MF_STRING : MF_GRAYED}});
menu.newCheckMenu(funct, 'Change SubMenu', 'Hola 3', () => {return (bSubMenu) ? 0 : 1;});
menu.newEntry({entryText: 'Hola 4', func: () => {console.log('hola4')}});

function on_mouse_rbtn_up(x, y) {return menu.btn_up(x, y);}

Renders to: (note 'Hola 4' Entry is drawn after the sub-menu)
+Hola
+-----
+HOla 2
+SubMenu 1 / SubMenu 2:
+Change SubMenu
+Hola 3
+Hola 4

- Manually adding some entries to the menu (uses previous code too):
var menuTwo = new _menuTwo();
function _menuTwo() {
this.idxInitial = 0;

this.btn_up = (idxInitial) => {
this.idxInitial = idxInitial;
const menuName = menu.getMainMenuName();
menu.getMenu(menuName).AppendMenuItem(MF_STRING, idxInitial + 1, 'Manual entry 1');
}

this.btn_up_done = (currIdx) => {
if (currIdx == this.idxInitial + 1) {
console.log('Manual entry 1');
return;
} else {return;}
}
}

function on_mouse_rbtn_up(x, y) {return menu.btn_up(x, y, menuTwo);}

Renders to:
+Hola
+-----
+HOla 2
+SubMenu 1 / SubMenu 2:
+Change SubMenu
+Hola 3
+Hola 4
+Manual entry 1
 */

include(fb.ComponentPath + 'docs\\Flags.js');

function _menu({bSupressDefaultMenu = true, idxInitial = 0} = {}) {
var menuArr = [];
var menuMap = new Map();
var entryArr = [];
var entryMap = new Map();
var idxMap = new Map();
var checkMenuMap = new Map();
var checkMenuArr = [];
var idx = idxInitial;

// To create new elements
this.newMenu = (menuName = 'main') => {
menuArr.push(menuName);
if (menuArr.length > 1) {entryArr.push({menuName: menuName, bIsMenu: true});}
return menuName;
}
this.newMenu(); // Default menu

this.newEntry = ({entryText = null, func = null, menuName = menuArr[0], flags = MF_STRING}) => {
entryArr.push({entryText: entryText, func: func, menuName: menuName, flags: flags, bIsMenu: false});
return entryArr[entryArr.length -1];
}

this.newCheckMenu = (menuName, entryTextA, entryTextB, idxFun) => {
checkMenuArr.push({menuName: menuName, entryTextA: entryTextA, entryTextB: entryTextB, idxFun: idxFun});
}

// Internal
this.getMenu = (menuName) => {return (!menuName) ? menuMap : menuMap.get(menuName);}
this.getIdx = (entryText) => {return (!entryText) ? entryMap : entryMap.get(entryText);}
this.getEntry = (idx) => {return (!idx) ? idxMap : idxMap.get(idx);}
this.getCheckMenu = (menuName) => {return (!menuName) ? checkMenuMap : checkMenuMap.get(menuName);}
// External
this.getNumEntries = () => {return entryArr.length;}
this.getMainMenuName = () => {return menuArr[0]}

this.createMenu = (menuName = menuArr[0]) => {
if (_isFunc(menuName)) {menuName = menuName();}
menuMap.set(menuName, window.CreatePopupMenu());
return menuMap.get(menuName);
}

this.addToMenu = ({entryText = null, func = null, menuName = menuArr[0], flags = MF_STRING}) => {
if (entryText == 'sep' || entryText == 'separator') {menuMap.get(menuName).AppendMenuSeparator();}
else {
idx++;
if (_isFunc(menuName)) {menuName = menuName();}
if (_isFunc(flags)) {flags = flags();}
if (_isFunc(entryText)) {entryText = entryText();}
menuMap.get(menuName).AppendMenuItem(flags, idx, entryText);
entryMap.set(entryText, idx);
idxMap.set(idx, func);
}
}

this.checkMenu = (menuName, entryTextA, entryTextB, idxFunc) => {
checkMenuMap.set(menuName, () => {
if (_isFunc(menuName)) {menuName = menuName();}
if (_isFunc(entryTextA)) {entryTextA = entryTextA();}
if (_isFunc(entryTextB)) {entryTextB = entryTextB();}
return menuMap.get(menuName).CheckMenuRadioItem(this.getIdx(entryTextA), this.getIdx(entryTextB), this.getIdx(entryTextA) + idxFunc());
});
}

this.btn_up = (x, y, object) => {
// Init menus
menuArr.forEach( (menuName) => {
this.createMenu(menuName);
});
// Add entries
entryArr.forEach( (entry) => {
if (!entry.bIsMenu) { // To main menu
this.addToMenu({entryText: entry.entryText, func: entry.func, menuName: entry.menuName, flags: entry.flags});
} else { // And append sub-menus
const subMenuName = _isFunc(entry.menuName) ? entry.menuName() : entry.menuName;
if (subMenuName != menuArr[0]) {
this.getMenu(subMenuName).AppendTo(this.getMenu(menuArr[0]), MF_STRING, subMenuName)
}
}
});
// Init checks
checkMenuArr.forEach( (check) => {
this.checkMenu(check.menuName, check.entryTextA, check.entryTextB, check.idxFun);
});
this.getCheckMenu().forEach( (func) => {
func();
});
// Call other object's menu creation
if (object && object.hasOwnProperty('btn_up')) {
object.btn_up(this.getNumEntries()); // The current num of entries may be used to create another menu
}
// Find currently selected item
const currIdx = this.getMenu(menuArr[0]).TrackPopupMenu(x, y);
let bDone;
this.getEntry().forEach( (func, entryIdx) => {
if (entryIdx == currIdx) {
func();
return bDone = true;
}
});
// Call other object's menu selection
if (!bDone && object && object.hasOwnProperty('btn_up_done')) {
object.btn_up_done(currIdx);
}
// Clear all
this.clear();
return bSupressDefaultMenu;
}

this.clear = () => {
menuMap.clear();
entryMap.clear();
idxMap.clear();
checkMenuMap.clear();
idx = 0;
}
}

// Helper
function _isFunc(obj) {
  return !!(obj && obj.constructor && obj.call && obj.apply);
};

I have written some examples of use, and just 11 lines of code
Code: [Select]
			var menu = new _menu();
menu.newEntry({entryText: 'Hola', func: () => {console.log('hola')}});
menu.newEntry({entryText: 'sep'});
menu.newEntry({entryText: 'Hola2', func: () => {console.log('hola2')}});
var bSubMenu = true;
const funct = () => {return (bSubMenu) ? 'SubMenu 1' : 'SubMenu 2';};
menu.newMenu(funct);
menu.newEntry({menuName: funct, entryText: 'Change SubMenu', func: () => {bSubMenu = !bSubMenu}});
menu.newEntry({menuName: funct, entryText:'Hola 3', func: () => {console.log('hola3')}, flags: () => {return (bSubMenu) ? MF_STRING : MF_GRAYED}});
menu.newCheckMenu(funct, 'Change SubMenu', 'Hola 3', () => {return (bSubMenu) ? 0 : 1;});
menu.newEntry({entryText: 'Hola 4', func: () => {console.log('hola4')}});

function on_mouse_rbtn_up(x, y) {return menu.btn_up(x, y);}
Translate into this:
X

No need to say that maintaining a few lines where you can easily see what each entry of the menu does is easier than having the code scattered at multiple places. Also changing entries order is just a matter of moving/adding/removing lines, since idx are calculated on the fly. The script requires no additional helpers and is ready 'as is' to be included on any other script. If you want a right button menu and a left button menu, then you simply create two different _menu() objects, with their entries, and use them on the different associated callbacks

Now back to the playlist creation topic, this helper can be used to create arbitrary menus linked to other scripts (like playlist creation functions). Similar to the image at bottom at the previous reply.

Code: [Select]
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\top_tracks_from_date.js');
var menu = new _menu();
var year = new Date().getFullYear();
menu.newEntry({entryText: 'Top Tracks from ' + year, func: () => {do_top_tracks_from_date({playlistLength: 50,  year: year})}});
year--;
menu.newEntry({entryText: 'Top Tracks from ' + year, func: () => {do_top_tracks_from_date({playlistLength: 50,  year: year})}});
year--;
menu.newEntry({entryText: 'Top Tracks from ' + year, func: () => {do_top_tracks_from_date({playlistLength: 50,  year: year})}});

function on_mouse_rbtn_up(x, y) {return menu.btn_up(x, y);}
...
Translates into:
X

Full code is at top. Is wip, and at some point I will share it too on github along all scripts.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-18 12:54:03
And with a few more work, we can merge the other scripts too:
Code: [Select]
'use strict';
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\helpers\\menu_creator.js');

var menu = new _menu();
const playlistSize = 50;

function on_mouse_rbtn_up(x, y) {return menu.btn_up(x, y);}

// Top Tracks from year
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\top_tracks_from_date.js');
var menuName = menu.newMenu('Top Tracks from...');
{ // Scopes force function to use the right variable at execution
const selYear = new Date().getFullYear();
menu.newEntry({menuName: menuName, entryText: 'Top Tracks from ' + selYear, func: (year = selYear) => {do_top_tracks_from_date({playlistLength: playlistSize,  year: year})}});
}
{
const selYear = new Date().getFullYear() - 1;
menu.newEntry({menuName: menuName, entryText: 'Top Tracks from ' + selYear, func: (year = selYear) => {do_top_tracks_from_date({playlistLength: playlistSize,  year: year})}});
}
{
const selYear = new Date().getFullYear() - 2;
menu.newEntry({menuName: menuName, entryText: 'Top Tracks from ' + selYear, func: (year = selYear) => {do_top_tracks_from_date({playlistLength: playlistSize,  year: year})}});
}
menu.newEntry({menuName: menuName, entryText: 'sep'});
menu.newEntry({menuName: menuName, entryText: 'Set year... ', func: () => {
const selYear = new Date().getFullYear();
const input = Number(utils.InputBox(window.ID, 'Enter year', window.Name, selYear));
if (!Number.isSafeInteger(input)) {return;}
do_top_tracks_from_date({playlistLength: 50,  year: input})
}});
menu.newEntry({entryText: 'sep'});
// Similar by...
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\search_similar_by.js');
var menuName = menu.newMenu('Search similar by...');
{ // Scopes force function to use the right variable at execution
const selSimilarBy = {mood: 6};
menu.newEntry({menuName: menuName, entryText: 'Moods', func: (similarBy = selSimilarBy) => {do_search_similar_by(playlistSize, undefined, undefined, undefined, similarBy, undefined)}});
}
{
const selSimilarBy = {genre: 2};
menu.newEntry({menuName: menuName, entryText: 'Genres', func: (similarBy = selSimilarBy) => {do_search_similar_by(playlistSize, undefined, undefined, undefined, similarBy, undefined)}});
}
{
const selSimilarBy = {style: 2};
menu.newEntry({menuName: menuName, entryText: 'Styles', func: (similarBy = selSimilarBy) => {do_search_similar_by(playlistSize, undefined, undefined, undefined, similarBy, undefined)}});
}
menu.newEntry({entryText: 'sep'});
// Same Style
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\search_same_style.js');
menu.newEntry({entryText: 'Same Styles', func: () => {do_search_same_style(playlistSize)}});
// Same Style & Moods
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\search_same_style_moods.js');
menu.newEntry({entryText: 'Same Styles and Moods', func: () => {do_search_same_style_moods(playlistSize)}});

X

Will continue working on it and adding all scripts.
Title: Re: [SMP] Music Graph development
Post by: MordredKLB on 2021-03-19 02:39:42
That's what I did.
My "problem" was %lastfm_played_times_js% was empty but %played_times_js% was not (i don't use last fm!). So my reasoning was... that last fm variable only had last fm's plays and the other variable had foobar's plays. That's why I merged them as is  (without checking duplicates).
If that's not right according to your info, then there is a bug somewhere. (or things work different for people without last fm component?) For sure, plays within foobar do not update the last fm variable too:
The issue is that %played_times_js% will also include any plays scrobbled by your same foobar instance. You'll need to attempt to remove duplicates, otherwise you'll double count plays for anyone scrobbling from foobar. You also need do things a little fuzzy because the exact seconds might not match. That's why removing anything within 30 seconds of the last play is a good idea (since you can't scrobble any track less than 30 seconds long).
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-19 09:04:36
That's what I did.
My "problem" was %lastfm_played_times_js% was empty but %played_times_js% was not (i don't use last fm!). So my reasoning was... that last fm variable only had last fm's plays and the other variable had foobar's plays. That's why I merged them as is  (without checking duplicates).
If that's not right according to your info, then there is a bug somewhere. (or things work different for people without last fm component?) For sure, plays within foobar do not update the last fm variable too:
The issue is that %played_times_js% will also include any plays scrobbled by your same foobar instance. You'll need to attempt to remove duplicates, otherwise you'll double count plays for anyone scrobbling from foobar. You also need do things a little fuzzy because the exact seconds might not match. That's why removing anything within 30 seconds of the last play is a good idea (since you can't scrobble any track less than 30 seconds long).
I think I'm really not understanding their usage at all. Any of these is right? (lastfm statistics being any last fm play, within web + scrobbles)

A)  %played_times_js%  contains both, foobar (1) and lastfm statistics (2+3), and %lastfm_played_times_js% contains only lastfm statistics but with some temp offset (2+3).
B)  %played_times_js%  contains both, foobar (1)  and lastfm scrobbles (3), done within foobar and %lastfm_played_times_js% contains lastfm statistics (2+3) (with some temp offset).

In any case, what's the reasoning of having duplicate data at all? Why are they not being filtered using the SDK? Or why not exposing a global variable having all without duplicates?
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-19 12:17:33
Following the dynamic menu creation (https://hydrogenaud.io/index.php?topic=120394.msg994925#msg994925), some updates. Have merged all buttons from here (https://hydrogenaud.io/index.php?topic=120394.msg992558#msg992558) and converted them into menus, adding some more.



Left to do:
-'similar artist' using last.fm data. Instead of using marc's auto-playlist approach which only created playlist with 1 artist (clickable), the idea is to merge the entire list and give a playlist with tracks from all of them.
-More special playlists
  +Anti-influences
  +Harmonic mixing with other parameters
  +Progressive Playlist with influences
-More predefined playlists using graph, etc.
  +No weights, just by graph
  +Configure scattering instrumentals, etc. (with radius checks)
Title: Re: [SMP] Music Graph development
Post by: MordredKLB on 2021-03-19 17:41:27
A)  %played_times_js%  contains both, foobar (1) and lastfm statistics (2+3), and %lastfm_played_times_js% contains only lastfm statistics but with some temp offset (2+3).
B)  %played_times_js%  contains both, foobar (1)  and lastfm scrobbles (3), done within foobar and %lastfm_played_times_js% contains lastfm statistics (2+3) (with some temp offset).

In any case, what's the reasoning of having duplicate data at all? Why are they not being filtered using the SDK? Or why not exposing a global variable having all without duplicates?
B is mostly accurate.
%played_times_js% ONLY includes plays from foobar. It doesn't know or care about last.fm scrobbles, although some or all of these plays could have also been scrobbled.
%lastfm_played_times_js% ONLY includes scrobbles from last.fm, and doesn't know or care where those scrobbles came from and some or all of them could have been from foobar.

foo_enhanced_playcount originally didn't retrieve scrobbles, and it was added a little later on. The reason the lists are different and can contain duplicates is because it's impossible to be 100% accurate with them. Let's say you start playing a 4 minute long song at 12:00:01. After 60 seconds, foobar records a play time stamp at 12:01:01.342 (it's down to the microsecond at least). At the 2 minute mark (50% through) your song is scrobbled to last.fm with a timestamp of when the song STARTED playing... i.e. 12:00:01 (only down to the second). At some point relatively early on I started adding 60seconds to the timestamps I received from last.fm so that they would hopefully be very close, but if you pull up the properties page of a song you'll see that very often last.fm last played will be 12:01:01 and foobar's will be 12:01:02. Still, that'd probably be somewhat easy to try and filter out most duplicates.

The problem is now what happens if 30 seconds into the song I started playing at 12:00:01 I pause the track for exactly 10 minutes while I'm on a phone call, then I unpause and continue listening to the whole thing. last.fm receives a scrobble for 12:00:01 when the song started, and the component returns it as 12:01:01 like always. However, foobar's recorded played time is now 12:11:01. I've still got a duplicate.

I decided to just throw my hands up and put the onus on the user how they want to handle it. Maybe that's not the best decision but it's the one that was made. I could add a %played_times_combined% or change the default behavior and attempt to filter out duplicates, but I don't have plans for that at the moment. I honestly don't know if anyone besides you or me is even using the actual played times for anything currently. I think the lastfm play counts and the %first_played_enhanced%/%last_played_enhanced% are much more common.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-19 18:53:55
Ok! Now I get it. Yep, I supposed the problem was that. Although I think %played_times_combined% should be provided by the plugin for consistency and simplicity.

Meanwhile, since the "fix" is not so difficult I will simply combine them following your advice of discarding plays differing by seconds. I don't use scrobbles at all so using only %played_times_js%  would be good enough for me, but thinking again about it... your plugin is mostly used by people who use lastfm, so it makes sense to use both features even if it requires a bit more of work.

Quote
I honestly don't know if anyone besides you or me is even using the actual played times for anything currently
Honestly, being so easy to add thinks using js to Foobar, I plan to expand the scripts more to add complex UI/statistics features (specially if SMP is developed with those aims in mind).

This was just a "top tracks" playlist generation script, but I could add anything seen on web players... like music recommendations using played tracks history, weighting tracks for inclusion on other playlist generation scripts, etc. the sky is the limit. So I see some potential on having %played_times_combined% for simplicity.

Also it would be great to update the entire library on demand with your plugin using a menu option or a popup after installation  (as far as I know the variables are only updated after playing the track for the first time).

And maybe adding 'artificial' play counts to last date, if first play count and last play count were from same year/month. I know you can't retrieve the real date, but that would at least preserve some info about the most aprox. date. Otherwise, on next play, the variables are updated, that info is lost and playcount and js variable differ in size.
Code: [Select]
playcount: 9
firstPlay : 2011-Sep
lastPlay: 2011-Dec
--> We can be sure those 9 playcounts must be from Sep to Dec.

Installation. After playing it now, with your plugin:
playcount: 10
firstPlay : 2011-Sep
lastPlay: 2021-Mar
playcount_js [2011-Sep, 2011-Dec,2021-Mar]
--> Previous info is lost.

My suggestion:
playcount: 10
firstPlay : 2011-Sep
lastPlay: 2021-Mar
playcount_js [2011-Sep, 2011-Dec, 2011-Dec, , ...8 times more on Dec ...., 2021-Mar]
--> Info is at least partially accurate for any date previous to installation

Thanks

PD: answered the pms, sorry for the delay. That's why I use the email
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-24 12:43:08
Some updates about merging all playlist tools.
- Added library tag checking tool script and entry on menu (tools). Checks all tag values from selected tracks for spelling errors or misplacing values in wrong tags, and reports identified errors (true positives), other posible errors, and possible fix. See this:  https://hydrogenaud.io/index.php?topic=120738.msg995165#msg995165
(https://hydrogenaud.io/index.php?action=dlattach;attach=19310;image)

- Some config and variables are saved as properties to be maintained on script reload. Global variables are configurable.


- Menus are created dynamically according to the components or scripts installed. i.e. without playcount_enhanced component, the related entries are not created and are substituted with query functionality. (https://hydrogenaud.io/index.php?topic=115227.msg994693#msg994693) The same for scripts, if something is not found, then that menu is not created.
- Added multiple greyed headers entries explaining the menus.
- Multiple bugfixes.
- Added multiple new special playlists.
- Moved remove duplicates and filtering to Tools submenu (along check tags).
- Separated config for remove duplicates and filtering, now they use different set of tags (configurable).
- Created a button to call the menu. Can be integrated with bars, and other buttons using the buttons framework.
- Added a tooltip function which shows info from focused track (the one to be used by the menus).


Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-26 21:16:09
Just sent an email to anyone who wrote me at some point with all scripts. Have tried to solve all bugs and make things as easy as possible. All previous reports for the playlist manager were fixed too.
If anyone is interested please drop me a line
regorxxx@protonmail.com

Left to do:
- Waiting for WilB's response about notifying tags on playback (for world map integration).
- Waiting for TheQwertiest's response about SMP methods to work with fpl (for playlist manager and Search by distance)
- Waiting for MordredKLB's response about play count future (for top tracks per year)

As soon as those things get solved, I think they are ready for public release.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-03-29 16:19:54
New update:
- Contextual menu to find playlists where the selected tracks reside in.
- Contextual menu to remove selected tracks from playlist where they reside in.
- Locked playlists are grayed on remove menu (but shown).
- Current playlist is grayed on find menu (but shown).
- See this: https://hydrogenaud.io/index.php?topic=120761.msg995424;topicseen#new
- In any case, only playlist with the tracks are shown; contrary to legacy command, which shows all and gives no indication about playlist having the tracks or not... being totally useless then.
X

- Splits list at 10 or 20 elements (> 100) per sub-menu if more than 10 playlists are to be shown. (fixes the UI cluttering of the legacy command with infinite entries shown at the same time) (for demonstration purpose, here it's split at 2 elements)
(https://hydrogenaud.io/index.php?action=dlattach;attach=19377;image)

Have not made any other big change, so I can send new links now without further testing. If anyone is interested please drop me a line
regorxxx@protonmail.com
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-01 21:43:46
regor I have this error :
Error: Spider Monkey Panel v1.4.1 (Playlist Generation Menu: Playlist Generation Menu by xxx)
GetQueryItems failed:
Invalid filter expression

File: search_same_by.js
Line: 230, Column: 29
Stack trace:
  do_search_same_by@search_same_by.js:230:29
  func@playlist_tools_menu.js:160:120
  _menu/this.btn_up/<@menu_xxx.js:222:5
  _menu/this.btn_up@menu_xxx.js:220:19
  newButtons.menuButton<@<main>:56:8
  SimpleButton/this.onClick@buttons_xxx.js:134:31
  on_mouse_lbtn_up@buttons_xxx.js:211:17

This happens when I do this : Playlist Tools - Special Playlists - Music by same composer(s) as artist(s)

This is the selected file that makes this error to appear :
Brad Fiedel - It's Over (Remastered 2017).flac
Album: Terminator 2: Judgment Day (Remastered 2017)
Genre: Filme/Jocuri video; Muzică de film

Also, that photo that you posted... on the right of the album art, where you have :
Field             Value
Metadata   
Artist           A flor de piel

What component is that ? Or is it a script?

Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-06 19:02:06
regor I have this error :
Error: Spider Monkey Panel v1.4.1 (Playlist Generation Menu: Playlist Generation Menu by xxx)
GetQueryItems failed:
Invalid filter expression

File: search_same_by.js
Line: 230, Column: 29
Stack trace:
  do_search_same_by@search_same_by.js:230:29
  func@playlist_tools_menu.js:160:120
  _menu/this.btn_up/<@menu_xxx.js:222:5
  _menu/this.btn_up@menu_xxx.js:220:19
  newButtons.menuButton<@<main>:56:8
  SimpleButton/this.onClick@buttons_xxx.js:134:31
  on_mouse_lbtn_up@buttons_xxx.js:211:17

This happens when I do this : Playlist Tools - Special Playlists - Music by same composer(s) as artist(s)

This is the selected file that makes this error to appear :
Brad Fiedel - It's Over (Remastered 2017).flac
Album: Terminator 2: Judgment Day (Remastered 2017)
Genre: Filme/Jocuri video; Muzică de film

The problem is the query created. You must check the console logs to see what the problem was (?) I can not give you any more info about it, without the query. The query is logged before executing it, so it will be present even if it crashes. It may be a problem with the forced query you set or a specific problem with that track.

For sure, I can execute the menu without problems with the latest release on tracks with or without composer tags (maybe you are using a previous version? I fixed something related on last release). Anyway I will warn with a popup instead of a crash when the created queries are invalid on next releases. I already do that for forced queries, but created queries should never be wrong (that was the point).

For ex, a track with composer tags: Michael Kiwanuka; Dean Josiah Cover
On console:
Quote
[20:01:30] Playlist created: (artist IS Michael Kiwanuka OR artist IS Dean Josiah Cover) AND NOT (%rating% EQUAL 2 OR %rating% EQUAL 1) AND NOT (STYLE IS Live AND NOT STYLE IS Hi-Fi) AND %channels% LESS 3 AND NOT COMMENT HAS Quad

The plugin is Item Properties panel from Columns UI (I think it's a panel from the UI, not another plugin).
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-07 00:09:18
It seems I wasn't using the latest scripts, I posted the message on 1, April and the email with the new scripts arrived in 31, March , and I wasn't aware of that, sorry I didn't checked the emails in that day. BUT , even with the new scripts, I have the same error, this is the console log :
Playlist created: (artist IS ) AND NOT (%rating% EQUAL 2 OR %rating% EQUAL 1) AND NOT (STYLE IS Live AND NOT STYLE IS Hi-Fi) AND %channels% LESS 3 AND NOT COMMENT HAS Quad
foo_spider_monkey_panel:
Error: Spider Monkey Panel v1.4.1 (Playlist Tools Menu: Playlist Tools Menu by xxx)
GetQueryItems failed:
Invalid filter expression

File: search_same_by.js
Line: 230, Column: 29
Stack trace:
  do_search_same_by@search_same_by.js:230:29
  func@playlist_tools_menu.js:161:120
  _menu/this.btn_up/<@menu_xxx.js:221:5
  _menu/this.btn_up@menu_xxx.js:219:19
  newButtons.menuButton<@<main>:56:8
  SimpleButton/this.onClick@buttons_xxx.js:134:31
  on_mouse_lbtn_up@buttons_xxx.js:211:17

It seems this might be the problem "(artist IS )"?
I don't have the composer tag for this file , BUT I tried on other files without composer tag and no crash happens.
Also the file is not corrupt ( I verified with foobar component, File Integrity Verifier and also other tools)
For me this is not a big problem, because until now this error happened only for this file but I thought you should know.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-07 00:36:07
It seems I wasn't using the latest scripts, I posted the message on 1, April and the email with the new scripts arrived in 31, March , and I wasn't aware of that, sorry I didn't checked the emails in that day. BUT , even with the new scripts, I have the same error, this is the console log :
Playlist created: (artist IS ) AND NOT (%rating% EQUAL 2 OR %rating% EQUAL 1) AND NOT (STYLE IS Live AND NOT STYLE IS Hi-Fi) AND %channels% LESS 3 AND NOT COMMENT HAS Quad
foo_spider_monkey_panel:
Error: Spider Monkey Panel v1.4.1 (Playlist Tools Menu: Playlist Tools Menu by xxx)
GetQueryItems failed:
Invalid filter expression

File: search_same_by.js
Line: 230, Column: 29
Stack trace:
  do_search_same_by@search_same_by.js:230:29
  func@playlist_tools_menu.js:161:120
  _menu/this.btn_up/<@menu_xxx.js:221:5
  _menu/this.btn_up@menu_xxx.js:219:19
  newButtons.menuButton<@<main>:56:8
  SimpleButton/this.onClick@buttons_xxx.js:134:31
  on_mouse_lbtn_up@buttons_xxx.js:211:17

It seems this might be the problem "(artist IS )"?
I don't have the composer tag for this file , BUT I tried on other files without composer tag and no crash happens.
Also the file is not corrupt ( I verified with foobar component, File Integrity Verifier and also other tools)
For me this is not a big problem, because until now this error happened only for this file but I thought you should know.

Yep, obviously the problem is "(artist IS )". In this case, your file probably has a composer tag but its value is a blank space, some strange char or simply empty (null tag?). A blank space is a valid tag value, but on queries that translates into an error. On future versions, the script will not crash but give an error popup. So while your problem is not a bug with the scripts but an error on your files, it will at least warn about those issues without crashing.

Btw, the new scripts have a tools/check tags menu. Try it on your entire library and it will probably report that error and some more ;)
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-07 17:33:01
New update on Playlist tools:

- Contextual menu to find playlists where the current playing track resides in. Also indicates which one is the playing playlist. Obviously clicking on the playing playlist is equivalent to 'Active now playing' foobar's menu, the difference here is that you can find the track in other playlist too (or find the playing playlist name without activating it).


- Contextual menu to filter playlists by query: A predefined set of queries is given, but they can be added or removed using the menus. These options complement the filtering by tags (usually used to remove duplicates), allowing to filter a playlist in any imaginable way.


- Added multiple config menus for all tools.

- Find in/ remove from Playlists menus can be set to only be created when selecting a max. number of tracks. Why? Because selecting 100 tracks means trying to find them, one by one, in all your playlists... which can slow down the UI when creating the button. Common use case is trying to find 1 or a few tracks in different playlists, not too many. This option solves the problem.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-07 17:45:37
New update on check_library_tags:
- Identified errors like empty tags, blank spaced tags (any number) & multivalued tags not split are always found and reported first, for the entire list of tracks selected. (will report problems like this: https://hydrogenaud.io/index.php?topic=120394.msg995807#msg995807)
- Then other errors according to the freq. threshold of apparition and max number of tag values per tag to report are shown.
- Some speed improvements and minor fixes. Takes 1-2 secs for 70 K tracks.


- Added an option to check tags against an user configurable dictionary. de_DE, en_GB, en_US & fr_FR are provided for demonstration, and en_US set as default. Other Hunspell-style dictionaries can be added. Obviously, checking tags against a dictionary is slower than any other method (+30 secs), so it's only done if it's configured at the properties panel. (planned to add configs as menus too)
-  The dictionary method also suggest some alternative terms. Right now the alternatives are unfiltered. Although pretty good, I plan to fine tune them showing only those that pass some threshold according to their Levenshtein distance. (85% like the other suggestions found when comparing tag values among them)


Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-08 01:14:01
regor, thanks for the message and for your help. Yes, I tried tools/check tags menu (and I replaced the code from check_library_tags.js as you said) but it didnt worked. The file doesn't have a composer tag or the value is a blank space, some strange char . This is how I solved the issue :
Solution 1- I converted the file (actually the entire album because the error appeared for all the files from this album) in flac again ( I tried in mp3 also), keeping the tags the same without any modification , problem solved no more errors, the script doesnt crash.
Solution 2 - I noticed the files don't have BPM in the tags so I used Serato to get the BPM ---problem solved without quality loss, no more errors. This was my fav solution , because I didn't wanted to convert the files again. What is strange is that I tried to add manually the value for BPM, but that error doesn't disappear. I even tried to add the BPM with the foobar plugin for BPM but the result was the same, the error was still there. I don't know what Serato did, but it did a good job :) Maybe is not tag related?

A few things about BPM and Key . I tried to do a comparison between MusicBrainz Picard and Serato, about the accuracy of detecting BPM and Key. From what I tested both are doing a great job (obviously is not 100% accurate) without significand differences in results.
A few differences exists:
Advantages for Serato ; the speed, the values for the tags are obtained much quickly
Disadvantages for Serato ; it's not free. Also the value from Date tag is modified after analyzing the files : for example 1991-01-01 it becomes : 1991 ( the day and the month disappear) .You can solve this by copying the values from the Date tag in another tag , before analyzing the files (obviously not a perfect solution)
Advantages for MusicBrainz Picard : It's free
Disadvantages for MusicBrainz Picard : it's not so fast like Serato , it involves a few more steps to do, to obtain the values for the tags.
Also for the BPM these deserve to be also mentioned :  foo_beatit https://hydrogenaud.io/index.php?topic=104631.0 and foo_bpm
https://www.foobar2000.org/components/view/foo_bpm
Let's say for example that both Serato and MusicBrainz Picard have obtained a wrong value for BPM, you can use one of these foobar components to analyze the file or you have the option to do it manually (Manually tap BPM) , which is great (MusicBrainz Picard can't do that)

Many days ago I wanted to ask you about a problem that Foobar has : the lack of a contextual menu to find ONLY the playlists where the selected tracks reside in . Fortunately you were much quicker and you solved the problem before I even had the chance to ask. So, huge congrats for this :) I use Foobar from 2005, I have also used Winamp, Aimp, Jriver,  and a few years ago I used MusicBee for a year. before I returned to Foobar again, MusicBee had this feature, contextual menu to find ONLY the playlists where the selected tracks reside in, so it was a bit frustrating that this wasn't possible in foobar. But now it is, thanks to you. Also I had the values for the Key tag but they were useless in audio players (they were useful only in dj programs like Serato, where you can do Harmonic Mixing) . More than a year ago MusicBee released a plugin that finally used the Key tag , so obviously your scripts are a very very good news for foobar, because it adds features that foobar was lacking , making it an even more complete audio player

Now a few suggestions (if it's possible and if you like the ideas):
1.Is it possible to have a component or SMP script that can replace this :  https://wiki.hydrogenaud.io/index.php?title=Foobar2000:Playlist_Revive_(foo_playlist_revive) ? The component does a great job BUT there is a problem : when you change the drive letter (lets say from D to G) then the component is useless:  dead items in a playlist are not replaced  with the matching ones in media library.
2 A script for SMP:  Spectrogram Seekbar? https://pastebin.com/6n0ZmTtB  This one is for WSH panel it requires SoX and ffmpeg to analyze the files. In CUI is not possible to add to the layout the Spectogram , only in DUI.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-08 10:13:13
regor, thanks for the message and for your help. Yes, I tried tools/check tags menu (and I replaced the code from check_library_tags.js as you said) but it didnt worked. The file doesn't have a composer tag or the value is a blank space, some strange char . This is how I solved the issue :
Solution 1- I converted the file (actually the entire album because the error appeared for all the files from this album) in flac again ( I tried in mp3 also), keeping the tags the same without any modification , problem solved no more errors, the script doesnt crash.
Solution 2 - I noticed the files don't have BPM in the tags so I used Serato to get the BPM ---problem solved without quality loss, no more errors. This was my fav solution , because I didn't wanted to convert the files again. What is strange is that I tried to add manually the value for BPM, but that error doesn't disappear. I even tried to add the BPM with the foobar plugin for BPM but the result was the same, the error was still there. I don't know what Serato did, but it did a good job :) Maybe is not tag related?

Well if the files are corrupted in some way, then that's beyond the scripts capabilities... sure foobar has some built in tools to check for malformed tags (that's not the point of my script anyway, which only checks their values).

Quote
A few things about BPM and Key . I tried to do a comparison between MusicBrainz Picard and Serato, about the accuracy of detecting BPM and Key. From what I tested both are doing a great job (obviously is not 100% accurate) without significand differences in results.
A few differences exists:
Advantages for Serato ; the speed, the values for the tags are obtained much quickly
Disadvantages for Serato ; it's not free. Also the value from Date tag is modified after analyzing the files : for example 1991-01-01 it becomes : 1991 ( the day and the month disappear) .You can solve this by copying the values from the Date tag in another tag , before analyzing the files (obviously not a perfect solution)
Advantages for MusicBrainz Picard : It's free
Disadvantages for MusicBrainz Picard : it's not so fast like Serato , it involves a few more steps to do, to obtain the values for the tags.
Also for the BPM these deserve to be also mentioned :  foo_beatit https://hydrogenaud.io/index.php?topic=104631.0 and foo_bpm
https://www.foobar2000.org/components/view/foo_bpm
Let's say for example that both Serato and MusicBrainz Picard have obtained a wrong value for BPM, you can use one of these foobar components to analyze the file or you have the option to do it manually (Manually tap BPM) , which is great (MusicBrainz Picard can't do that)

Right. I simply mention Picard as one utility that can do it, but there are many more. I also use one of those foobar plugins to manually edit BPM from time to time as you noted. Its your choice to use the tool that suits your needs. Anything that gives a BPM is good enough (since tags can be mass renamed later).

Quote
Now a few suggestions (if it's possible and if you like the ideas):
1.Is it possible to have a component or SMP script that can replace this :  https://wiki.hydrogenaud.io/index.php?title=Foobar2000:Playlist_Revive_(foo_playlist_revive) ? The component does a great job BUT there is a problem : when you change the drive letter (lets say from D to G) then the component is useless:  dead items in a playlist are not replaced  with the matching ones in media library.
Have to look at it, but if my understanding of playlists is right, then it should be pretty trivial to do it with a script. The playlists store the handle's path of every item, so I can easily find a match if the only thing that changes is the path. Doable.
I would also prefer to have it done via script, to reduce the number of components being loaded by an exe on windows (I'm already at the max which creates bugs on VST plugin loading).

You can also save the playlist as m3u, and edit it replacing drive letters. Or use an external tool like listfix (recommended):
https://sourceforge.net/projects/listfix/

Quote
2 A script for SMP:  Spectrogram Seekbar? https://pastebin.com/6n0ZmTtB  This one is for WSH panel it requires SoX and ffmpeg to analyze the files. In CUI is not possible to add to the layout the Spectogram , only in DUI.
I use CUI and I'm not sure what the problem is? You can not add multiple SMP bars to the menu bar (which I find annoying and I would suggest to change that). But that's a foobar's design decision.  I don't see why you can't add a new bar as an independent panel just below the menu bars (?) I mean, if that script is working in WSH, my own version will not do anything differently. You should ask marc2003 @snotlicker to update it for SMP though, if he has interest. I'm not so interested in UI scripts if they are just eye candy, so that would be really low in my priority list.
X
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-08 22:27:31
The playlists store the handle's path of every item, so I can easily find a match if the only thing that changes is the path.

But what if the drive letter isn't the only change? For example you move the files on another drive but also in different folders? Is this still possible? Thanks anyway, for trying to do something about this, when you'll have time.
Yes, I know about listfix, great tool indeed but obviously it will be awesome if this can be done in foobar without the need of another external tool (and also it will complement the other tools that you already have for playlists).
About a possible script for SMP:  Spectrogram Seekbar , Sorry, I wasn't clear enough. What I wanted to say is this. For those that use DUI , they already have the possibility to add a Spectrogram panel to the layout, for us on CUI this is not possible (sure, you can do this: View - Visualizations - Spectrogram . But it's only a popup panel, you can't add a panel with the Spectrogram to your layout). To solve the problem this script (Spectrogram Seekbar) for WSH panel exists.
But I understand your opinion, so no problem. Maybe snotlicker will try to do it. It's not entirely useless, to have a visual look about the quality of a song. and it looks great too :)
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-09 00:36:31
The playlists store the handle's path of every item, so I can easily find a match if the only thing that changes is the path.

But what if the drive letter isn't the only change? For example you move the files on another drive but also in different folders? Is this still possible? Thanks anyway, for trying to do something about this, when you'll have time.
Yes, I know about listfix, great tool indeed but obviously it will be awesome if this can be done in foobar without the need of another external tool (and also it will complement the other tools that you already have for playlists).
About a possible script for SMP:  Spectrogram Seekbar , Sorry, I wasn't clear enough. What I wanted to say is this. For those that use DUI , they already have the possibility to add a Spectrogram panel to the layout, for us on CUI this is not possible (sure, you can do this: View - Visualizations - Spectrogram . But it's only a popup panel, you can't add a panel with the Spectrogram to your layout). To solve the problem this script (Spectrogram Seekbar) for WSH panel exists.
But I understand your opinion, so no problem. Maybe snotlicker will try to do it. It's not entirely useless, to have a visual look about the quality of a song. and it looks great too :)
Finding the same track at different folders is possible but requires some guessing. If all tags are stored in database and you did not change them, then it's trivial with 100% accuracy. Otherwise, the tool could replace a missing track with another "similar" enough track, but I would prefer 100% accuracy whenever it's possible.

Ok now I get your point. You are right. But anyway I'm only working on SMP scripts, so creating a component for such panel is out of my list.
It can be done on SMP, but being already done on WSH, it's not a priority for me since you can install both components and integrate it into a  WSH panel. And then snotlicker should be the one updating his scripts, but if I find time to port it at some point and it's easy enough I will take care of it.

I would ask CUI's author in his thread about integrating DUI panels into CUI panels. It has been already suggested many times and it was in the ToDo list (I think). That would solve the issue along many others (for ex. facets being only available as popup too).

Right now I consider:
      - Revive dead items on playlist.
      - Reuse the sorting logic of Special Playlists to sort playlists already created by the user:  the idea is getting the best of both worlds, playlists manually created and intelligent sorting to make them sound "right"
            - Using harmonic mixing (already coded)
            - Incremental keys (they must be translated into Camelot wheel notation, already coded)
            - Scattering instrumentals (already coded)
            - BPM (can be done using foobar sorting, but since I will add the others, will add this one too to the menus for simplicity)
            - ...
      - Expand the world map code:
             -  Show statistics about the location of all artist on library.
             -  Merge near enough locations into "zones", to draw the points for multiple artists.
             -  Use the points to create playlists based on the zone selected with artist from that zone.
             -  ...
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-09 23:15:51
The playlists store the handle's path of every item, so I can easily find a match if the only thing that changes is the path.

But what if the drive letter isn't the only change? For example you move the files on another drive but also in different folders? Is this still possible? Thanks anyway, for trying to do something about this, when you'll have time.
Yes, I know about listfix, great tool indeed but obviously it will be awesome if this can be done in foobar without the need of another external tool (and also it will complement the other tools that you already have for playlists).
About a possible script for SMP:  Spectrogram Seekbar , Sorry, I wasn't clear enough. What I wanted to say is this. For those that use DUI , they already have the possibility to add a Spectrogram panel to the layout, for us on CUI this is not possible (sure, you can do this: View - Visualizations - Spectrogram . But it's only a popup panel, you can't add a panel with the Spectrogram to your layout). To solve the problem this script (Spectrogram Seekbar) for WSH panel exists.
But I understand your opinion, so no problem. Maybe snotlicker will try to do it. It's not entirely useless, to have a visual look about the quality of a song. and it looks great too :)
Finding the same track at different folders is possible but requires some guessing. If all tags are stored in database and you did not change them, then it's trivial with 100% accuracy. Otherwise, the tool could replace a missing track with another "similar" enough track, but I would prefer 100% accuracy whenever it's possible.
Got it working. Workflow is simple, gets the list of tracks (from selection or playlist) and finds the dead items.
Then applies a query to the library according to their title (to make it faster), and compares length and file size.
If an exact match is found, it stops, otherwise it continues with the tags (a track may have changed path and other tags).
If the new track's tags is a superset of the old tags, then it's a match. Otherwise, there is a similarity Threshold... (note the tracks are already filter by title, so a 50% similarity means changing other tags)

Have integrated into the menu, like the other tools. And added different options (on selection, on entire playlist or just simulate)
X

In any case, the results are explained on the console. On simulation, that's the only output (no changes made to playlist)
X
The one chosen is the one with the highest scoring (but on simulation, all possibilities are logged for info). And as always, it can be undo.

Would want to take away the pre-filtering by title at some point (to allow title changes too, replacing it with artist), but it's good enough by now, surpassing the original component functionality.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-10 17:27:46
.. was for SMP thread
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-15 09:01:12
New update on playlist_revive: (new script)
   Alternative to foo_playlist_revive.
   Playlist Revive makes dead items in a playlist alive again by replacing them with the matching ones in media library.
   A handy utility for those who often move or rename their media files/folders.
   
   Matching:
      - Audio MD5 (Exact Match)
      - Title + Length + Size (Exact Match)
      - Tags (Similarity)
   
   Usage:
      - Select the tracks in the relevant playlist.
      - Apply script (using a button, menu entry, main menu SMP, etc. associated to it).

(more images on previous replies)

Playlist tools:
- Integrated playlist revive.
   + On selected items.
   + On entire playlist.
   + On all playlist.
   + Simulate: outputs only to console without making changes.
   + Find only: Find dead items on all playlist and shows the playlist names.


(more images on previous replies)

World Map:
- Integrated with Biography 1.1.3 script'.
   + Requires mod version (given along my scripts) until it gets updated by its author.
   + Can write locale tags automatically if not present on local files.
- Menus to configure:
   + Following selection or Playback.
   + Enable / Disable.
   + Enable / Disable WilB's Biography integration.
(https://hydrogenaud.io/index.php?action=dlattach;attach=19455;image)

Check tags:
- Added even more error checks which are shown on the report if identified. The last batch should cover most of them.
- Added multiple comments and tips on the report which explain how to solve the errors found.
- Added queries (foobar search / mp3tag) and TF expressions (facets) auto-generated according to the errors found to easily identify the related tracks.


All scripts:
- Multiple bugfixes, Checks for queries to avoid crashes (show popup instead).
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-16 20:16:21
New updates on World Map:
- Integrated with Biography 1.1.3 script'.
   + Can write tags to json file instead of track's tags.
   + json data is used even when Biography script is disabled, being an offline database of the tags.
- Search Locale tag, by order, using these sources.
   + Track's tag.
   + json data
   + Biography Script (online)
- Menus to configure:
   + Writing tags (3 options)
- Mouse info:
   + On mouse over a point, a tool-tip is shown with the country name. Point changes color.
X

- AutoPlaylist creation:
   + On click over a point, an autoplaylist is created with any artist on your library from the selected country. Checks for the track's tags, the JSON local database and online data for current playing / selected artist (i.e. it uses the 3 sources if possible).
X

   + Configurable forced query is added too.  (like in all my playlist scripts)
- Info:
   + Added a popup on first execution explaining a bit how the panel works. (plan to add something similar to ALL scripts)
X

TODO:
   + Picard's Plugin to get the tags on batch
   + Autoplaylist creation alternative: Force same genre or style than current track on ctrl + left click. That way it would search any artist from same country and same genre/style, showing culturally related music.

New updates on Playlist tools:
- Added top_rated_tracks script.
   + rating tag and limits (like 1-4) can be configurable. "Top track rating" is automatically calculated with that, getting first higher value, and then lowering it until the playlist is full (configurable playlist size).
   + On menu, it's used along a year or a range of years. So you can easily create a playlist with the most rated tracks from the 70s or a configurable date (or range). (dates are automatically calculated using current year)
   + Configurable forced query added too.
X
X

- Added multiple errors checks on properties, popups, etc. Should be pretty error proof now.
- Added configurable menus, which can be added/removed by the user. Allowing to create your own entries and functionality. (defaults can always be restored)
X
X

- Menu is now enabled when there is no selected, focused or playing track. Any entry which needs a selected track is now greyed out. (previously I simply disabled the entire menu). That way you can use tools not related to a selection even when the active playlist is empty.
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-20 16:55:17
New updates on World Map:
- Works with multiple selected tracks (draws all points on the map), allowing to show statistics of an entire playlist or library.
- Mouse info:
   + Tool-tip shows the tags added at properties for playlist creation (ctrl modifier).
   + Tool-tip shows the number of tracks for a given country..
X

- AutoPlaylist creation:
   + On click over a point, an autoplaylist is created with any artist on your library from the selected country. It also checks for the any know artist from that country, so those artists are always matched (even if your files have no tags updated yet). Also uses the data from the JSON file, covering any possibility. The rationale is simple, if Ziggy Marley is from Jamaica, the query will look for any track with locale = Jamaica but also for any track with artist = Ziggy Marley... since that artist is always associated to the same country.
X

   + Ctrl modifier: forces an autoplaylist  with artist from selected country AND same tags (2 ,configurable). Currently set to style and genre. When selecting only 1 track is easy, the autoplaylist created forces same genre or style. When selecting multiple tracks, the auto-playlist must match any of the combinations of genres or styles of any given track. In other words, selecting a rock artist's track and a folk artist's track from USA would produce an autoplaylist with artists from USA and genre Rock or Folk.
X

- Config:
   + 2 tags for ctrl modifier autoplaylists:
X

   TODO:
      - Picard's Plugin does not exist yet.

Notes:
Obviously, something similar can be done with "Search same by tags" from previous post (https://hydrogenaud.io/index.php?topic=120394.msg996309#msg996309). Setting to match the last locale tag and same genre or styles.
But there are some differences:
- The map works even without tracks tagged (while the playlist tool creates the queries based on tags from the selection). Therefore the map works in more situations.
- Playlist creation is done visually selecting a point. To do the same with the other script, you would need to select specifically a track from X country first. With the map you can select the entire library and filter your country with a mouse click.
- The map approach uses all tags from all tracks selected (linked to the selected country), linking easily similar music from the same culture, i.e. you can create autoplaylist from Mali with X different genres just by selecting some of your favorite tracks from Mali. Doing the same with the other script requires a lot more of work.
- Obviously the map is an UI panel, the other is just a tool for playlist creation.
- And the map panel can also tags your files while the other tool is only meant to read and use them.
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-21 22:16:53
regor, we have Top Tracks from a year (let's say 2020), I think it's this script:  top_tracks_from_date.js ,  which is great already. But, can we also have Top tracks , from the last 14 days,1 month, 3 months, 6 months?
Btw, great job with Top rated tracks, and the world map.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-22 08:49:24
For simplicity the entry named "From year... " lets you select any year, but I don't allow adding a query like "DURING LAST 57 WEEKS" as you noted. I will change that to allow both inputs.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-22 10:23:08
Update on top_tracks_from_date:
- Allows a 'last' argument which lets you use time expressions (https://wiki.hydrogenaud.io/index.php?title=Foobar2000:Query_syntax#Time_expressions) instead of an specific year.
   + Only Weeks and Days are allowed.
   + Coded to be easily expandable to new time units if they are added later.
- Works the same than the standard use, but outputs most played tracks within that period.
- Checks if the component (foo_enhanced_playcount) is installed when called. Previously it was only checked at menu creation, now on the function too... since someone may call it at its own buttons or scripts, just for safety.

Update on playlist tools:

- Added a new input entry for the previous feature.

X

Obviously the new functionality requires foo_enhanced_playcount (https://www.foobar2000.org/components/view/) the same than using a year.

Now working on this:
X
search_bydistance.js has a feature which allows to scatter instrumental tracks after playlist creation, to prevent multiple instrumental tracks being in consecutive order. I have taken that base and coded it to be more general, allowing any tag and value to be checked. Just select the items on a playlist and they will be reordered according to the tags (not matched tracks being untouched)....

Next step is sorting by Key using Camelot Wheel notation. Since Keys are strings and not numbers, they can not be sorted using standard foobar functionality without translating the keys into something understandable. Again, the same script has already the code for that translation, so I will port it to allow sorting of any selection by keys (going from 1 to 12).

(https://pyramind.com/wp-content/uploads/2020/05/preview-lightbox-CamelotWheel-1.jpg)

Finally, I will implement as a standalone utility the harmonic mixing functionality (https://hydrogenaud.io/index.php?topic=120394.msg994472#msg994472). Since it will be used on an already generated playlist, it will only reorder the items to follow a random harmonic mix. Any track not able to follow the path will be deleted afterwards.

At that point I think all scripts will be more or less done and ready for public release on github. Although I'm still waiting for an SMP update to work with fpl playlists... (otherwise the playlist manager will be incomplete)
Title: Re: [SMP] Music Graph development
Post by: Koshingg on 2021-04-23 00:51:23
For simplicity the entry named "From year... " lets you select any year, but I don't allow adding a query like "DURING LAST 57 WEEKS" as you noted. I will change that to allow both inputs.
Thank you
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-26 10:51:35
Update on World Map:
- Previously, Biography integration required a modified version of the script (since the author has not updated it yet) I was supplying along my files. Now the integration is added via script too. There is a menu option to install the mod (it looks for the original file, edits the relevant lines and creates a backup). Is done step by step and can be reverted back. Therefore it's as easy as it can be and I don't need to provide the original file.
- Integration has been expanded and not the Biography panel exposes its entire properties lists to other panels.
- Selection mode changes automatically when changing it on Biography panel, therefore syncing the changes. (previously it would simply broke the sync functionality). A popup is shown to warn about it.
- Since World Map script allows drawing multiple points (therefore allowing multiple selections and not only the focused track) but Biography Panel allows only 1 track (now playing or focused one), handle lists on drawing are compared to only sync the locale tag for the relevant track. That solves the problem produced when selection mode was different in both panels. (anyway it has been fixed too with the sel mode syncing, this is just another check)
- Previous change allows to save the locale tag to files or json even if nothing is drawn on the map. Biography lets you write tags on demand, this lets you write tags as soon as the panel gets refreshed with new data.
- Some bug fixes.
X

Update on Playlist Tools:
- Added readme to most scripts.
- Added readme menu to show the readme of every utility included.
X
- Main readme is shown the first time the script is installed.
- Added sort by Key menu: sorting by Key using Camelot Wheel notation. Since Keys are strings and not numbers, they can not be sorted using standard foobar functionality without translating the keys into something understandable for foobar. The script does that first and  then sorts the playlist using TF.
- Added harmonic mixing menu: DJ-like playlist creation with key changes following harmonic mixing rules using the entire playlist or just the selection. Note something similar was already added on 'Special Playlists'. The main difference is that this menu uses only the current selection/playlist as pool, while the other used 1 track to find similar tracks using complex algorithms (and then used that pool to create the harmonic mix playlist).
X

Update on Search by Distance:
- Added readme. Shown the first time the script is installed.

Update on Playlist Manager:
- Added readme. Shown the first time the script is installed.
- Added menu to edit the playlists path. (instead of using properties panel).
X

Quote
At that point I think all scripts will be more or less done and ready for public release on github. Although I'm still waiting for an SMP update to work with fpl playlists... (otherwise the playlist manager will be incomplete)
This is the only thing missing now.

Meanwhile I will check all files, clean unused things and add comments or explanations to the readmes whenever it's needed for public release. Should be on github on a few days or weeks.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-04-28 09:00:53
Update on Playlist Tools:
- Added a reset config menu, to reset all properties related to the script. Why? Because some menus are saved as properties, so everytime  a new one is added as default and the script is updated, the panel needs a reset to show it. Having in mind the menu may be used along other buttons, that means clearing all properties or deleting 20+ entries one by one. Now that menu does it automatically only for the relevant properties. I will consider saving those things as json too on the future, to easily import or export user set menus.
X

- Added dynamic evaluation of queries on Query Filtering menu. i.e. "GENRE IS #GENRE#" translates into "GENRE IS ROCK" when selecting a rock track, "%date% IS #$year(%date%)#" would translate into "%date% IS 1969"  for a track from 1969, etc..
Search by same did something similar with tags combinations, but Query Filtering is used to show the tracks on the playlist which pass the query (instead of retrieving tracks from library) and more important, it allows arbitrary TF expressions within the "#" symbols.

X


Already created the github and wanted to know if people prefer to have all scripts within a single repository or they should be split into different ones. For sure some things should be uploaded together, but other big scripts like the playlist manager or the Search by Distance thing I think it's better to have them on its own repository.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-01 16:16:48
Just sent the last batch of scripts while I'm working on the github (https://github.com/regorxxx) thing and documentation.
Download here: https://we.tl/t-MryTpewR6q
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 03:00:05
World-Map-SMP can be found here:
https://github.com/regorxxx/World-Map-SMP
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 10:39:54
Camelot-Wheel can be found here: (it's only a js helper for other projects, the implementation for foobar will be in another repository)
https://github.com/regorxxx/Camelot-Wheel-Notation
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 11:13:37
Music-Graph can be found here: (js helper for other projects or visualization, foobar implementation in another repository)
https://github.com/regorxxx/Music-Graph
Title: Re: [SMP] Music Graph development
Post by: kgena_ua on 2021-05-02 12:05:26
regor
World-Map-SMP-main
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 12:32:45
You have installed it wrong. Read instructions
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 12:39:30
Search-by-Distance-SMP can be found here: (foobar implementation of Music-Graph)
https://github.com/regorxxx/Search-by-Distance-SMP
Title: Re: [SMP] Music Graph development
Post by: kgena_ua on 2021-05-02 13:33:36
You have installed it wrong. Read instructions
Quote
INSTALLATION:
-------------
Copy all files from the zip into YOUR_FOOBAR_PROFILE_PATH\scripts\SMP\xxx-scripts
Any other path WILL NOT work without editing the scripts.
For ex: mine is          c:\Users\xxx\AppData\Roaming\foobar2000\scripts\SMP\xxx-scripts\...
For portable installations: .\foobar2000\profile\scripts\SMP\xxx-scripts\...
Then load any script into a SMP panel within foobar. See info for usage.
what is wrong
Title: Re: [SMP] Music Graph development
Post by: Air KEN on 2021-05-02 14:17:38
Hi :)
@kgena_ua

profile folder migrating
https://hydrogenaud.io/index.php?PHPSESSID=f2iv700108t207jee068urmjs2&topic=119619.0
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 15:12:13
.
Title: Re: [SMP] Music Graph development
Post by: snotlicker on 2021-05-02 15:26:21
Your instructions are incomplete.

If people are running fb2k v1.4/1.5 they won't have a profile folder in portable mode.
If people are running v1.6 but it was upgraded continuously from an older version, they won't have a profile folder as the previous file/folder structure is not touched.
Only clean portable installs created with v1.6 or later are guaranteed to have a profile folder.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 15:51:04
Your instructions are incomplete.

If people are running fb2k v1.4/1.5 they won't have a profile folder in portable mode.
If people are running v1.6 but it was upgraded continuously from an older version, they won't have a profile folder as the previous file/folder structure is not touched.
Only clean portable installs created with v1.6 or later are guaranteed to have a profile folder.
That makes sense. I don't use portable at all, so I only added that part because someone reported it to me (who was probably using 1.6). Thanks.
I will edit the readme again then.

Code: [Select]
what is wrong 
For portable installs >= 1.6:
.\foobar2000\profile\scripts\SMP\xxx-scripts\...
X

For portable installs < 1.6:
.\foobar2000\scripts\SMP\xxx-scripts\...
X
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 17:14:03
Playlist-Tools-SMP can be found here: (foobar implementation of the entire collection of tools related to playlists. Includes Search-by-Distance-SMP)
https://github.com/regorxxx/Playlist-Tools-SMP

NOTE: changed the readme on all repositories too. Only left the playlist manager, which still needs an SMP update to work with fpl playlists.
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-02 21:08:56
Playlist-Manager-SMP can be found here: (pre-release)
https://github.com/regorxxx/Playlist-Manager-SMP
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-05 22:40:47
Menu-Framework-SMP can be found here: (js helper to easily create easy to maintain menus on other projects)
https://github.com/regorxxx/Menu-Framework-SMP
Title: Re: [SMP] Music Graph development
Post by: regor on 2021-05-05 22:45:15
Have updated several repositories with some changes and added some hotfixes (specially the playlist manager wich is a pre-release).
Next will be the Map-Framework-SMP (to create arbitray maps according to tags) & Buttons-Framework-SMP (like the menu framework, for buttons)
SimplePortal 1.0.0 RC1 © 2008-2021