Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: [SMP] Music Graph development (Read 9087 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

[SMP] Music Graph development

First my request: I'm looking for a few users to try and test a SMP collection of scrips I have been working on. When they are ready, I will public them on github and here, but right now I'm just looking for betatesting. This is my first time coding on JS and after +2000 lines of code some help would be appreciated testing it!

Back to the topic. Some time ago, while tinkering with foobar I found it was really difficult to create autplaylist based on similarity checks. Things like these:
  • Similar Artist
  • Date
  • Genre similarity
  • Style similarity
  • Key
  • BPM
  • Moods
  • Top 25 most played tracks

Things like Random Pools component, current SMP scripts or Search for same components (I know 2) have many limitations for those things:
  • Genre similarity: For example, you can create a random pool of Rock and Alt. Rock. And it works great! But... you can not create dynamically a random pool for the genre of the currently selected track. Random pools are meant to be pre-set. You would need to create a random pool for every genre on your library if you want those playlist....
  • Date: The same. You can put a date on random pools. But it's a static one. Every time you want to change the date of your pool, you have to reconfigure the pool.
  • Moods: any plugin it's either checking if Moods are equal or not (search for same). But you can not check 4 equal moods for ex.
  • Top 25 most played tracks: Random pools and other plugins can provide a top 25 tracks. But if you have tracks with same title but different album versions, they are duplicated! Foobar remove duplicates only remove the same exact tracks. You could have the same track appearing 3 times, rendering that playlist useless. There is no workaround for this with any plugin I know.
X

  • Similar Artist: Yep, we got Lastfm scripts BUT you can only create an auto-playlist with just 1 similar artist. Not all of them! The point to have similar artists auto-playlist is creating playlist with artist similar to the currently selected track, all of them. Not only one.. I want to select Led Zep, press a button and create a playlist with all those similar artists.

I could continue...  but essentially, foobar is missing the functionality spottily gives, or allmusic, or musicbee, etc. Essentially the ability to create playlist dynamically based on tags or internet info of the currently selected track. There are some things done on that field but really limited on functionality.

There is also a music server app name MusicIp which works similarly to the things mentioned. Short story: it requires days to scan 50k tracks, it's an unsupported program from 10 years ago,  requires the server to be loaded along foobar, CMD scripts to send tracks to server and load playlists AND... it doesn't support multi tags. So all auto-playlist created rely on the musical analysis, but tags become useless. Also rating is not loaded, you have to create scripts to load foobar ratings to the server, manually! The program accepts some scripting and advanced commands, but there is no documentation and it's closed source... so good luck with it. Btw, someday I will post the things I managed to make work properly (only thing missing is multi-tag support).

All these findings lead me to AI reasearch. I mean... MusicIp "works" but it uses musical analysis from a decade ago. Nowadays we have machine learning models giving us things like genre or moods. That's the thing Spotify, Allmusic or any other network player uses. So we have 2 problems to solve: first, finding how to analyze music using modern methods. And tag it! Second, use foobar to create dynamic playlist with that info!

The first one is partially solved using Musicbrainz Picard. You can get mood, BPM and key tags from that program. And any file missing those variables can be analyzed with an exe which uploads the analysis to the server; 1 day later you re-load those files in Picard and magic. The missing tags are there.
I have managed to tag  (mood, occasion, BPM, Key) 99% of my files with that method, while adding manually things like genre and style ("subgenre"). Things like similar artists are provided by the last.fm component (can be saved to local tags!) or SMP scripts.

Tagging is mostly done. but the second one, previously discussed, its unresolved up to now. We have 2 things to consider now: dynamic playlist creating and genre similarity, whatever that means. Back to AI research, I thought I could find something useful on that field... without luck. "Solutions" require complicate scripting, a range of tools and then finding a good set of reference... but someday I found this Music map. There have been other similar maps, most interactive, giving you youtube videos of the selected style.
X


So, considering automated music genre mapping was out of question, I thought that could be translated to code. At the same time I was beginning to discover SMP scripts, so I tried to translate that map to js. This is my fist time on js, so it has been a hard road.

My first attempt is what I call "DYNGENRE". It's a pretty simple concept: let those 18 Supergenres (following the web convention) be assigned a number from 0 to 18. Any genre tag or style tag should fall into one of those supergenres. We can simplify it even more and not make any difference between "modal jazz" or "cool Jazz", they are just jazz (9)! And assign to those styles which are a mix of some supergenres their set of numbers: "Trip-Hop" (12,18) would be "Hip-Hop" (12) and "Downtempo" (18).

That assigns any track to a Supergenre (or a mix) according to genre tags and styles, and the best part: it's done on the fly. No need to re-tag. No need to recheck files when changing scripts. The DYNGENRE map can be set as you wish just by tweaking the js file.

The other problem was dynamic playlist creation, but I managed to code that for the currently selected track. It takes the mood, BPM, Genre, Style, Date and key tags and create playlists based on those tags. That means no more statics pools. You select rock from the 90s, and it outputs things according to that directly to a playlist (bypassing the need to press 20 buttons using the "search for same" plugins).

While working on that, I got the "duplication tracks" problem again... essentially, if I looked for things similar to a  Led Zeppelin track then it would output things from Led Zep,  Deep Purple, etc. but If I had 4 versions of "You shook me"... then all 4 got selected on the final playlist! That was one of the biggest problems with other plugins too. I want a playlist with things appearing once, not the same track all around from different albums, compilations, live sets, etc. So I created some scripts to solve that too.

The "remove duplicates script" looks for tracks on the handle list or playlist with same title/artist/date and remove all but one. It can be configurable to any other tag too and maintains the order. The last part is important! The script greatly surpasses the built in foobar remove duplicates functionality and the one on SMP methods too.

Other things I added are the combination of multitags. i.e. "I want only tracks with 5 similar moods", but if the tracks have 8 different moods, then you have to create k-combinations (k=5) of the entire set and check for every combinantion via queries. That's impossible to do on static pools or manually. Then we have user adjustable weighting of tags, filtering of any track below a minimum score, pool selection by lowest distance or random, etc
X
 
With all the problems solved, the final result is a playlist generated based on the tags of currently selected track with no duplicates and following similarity rules. All user adjustable.

And that's the "DYNGENRE" method which would be a more complex version than the "WEIGHT" method. My collection of scripts, ready to use, offer that (search by distance), along remove duplicate scripts (ready to be used standalone on playlist), Top 25 Tracks (without the duplicates problem), search similar style + k-com of moods and Search for same style (multitag).

This is a screenshot of the thing working. I took a folk track, and 857 tracks were eligible on the pool (score over 65%). Final selection, 50 tracks.
X
X

Re: [SMP] Music Graph development

Reply #1
After all this discussion, here it comes the second part. Along the "DYNGENRE" and "WEIGHT" methods, there is a third one: "GRAPH". Remember the previous music map examples... I simplified the thing assigning main genres/styles to numbers and not much more! But what about a real map (graph) with genres and styles? This:
X

Well, after coding all the previous things I tried creating a real map. With distances and links. I found this js library which does exactly what I wanted: graphs, nodes, links and the most important thing, pathsearch. But it was for node.js. I find how the hell put that thing to work within foobar and got a graph created using arrays with all the genres and styles. And I mean all. Then adding supergenres, clusters, style clusters, etc.
X

I put all that on the "search by distance" script, configured some limits for the paths and got it working. Now "Coutry Blues" is not just "Blues" (dyngenre method), but a node within blues supergenre. And it would be more distant to Electric blues than to Delta blues, for ex. Then we got style clusters and primary origins/derivations to traverse between different super-genres.. and all that. It takes 2-5 seg to search 40k tracks and calculate paths and distances within foobar. It works, and creates playlists.

My final step on the development have been the graphical part. Coding +60 arrays with nodes and links is great but it becomes a crazy thing, to the point I don't know if I'm duplicating things or setting things wrong. And here the graphical representation comes to help. I managed to render the graph within firefox using other JS library and creating an HTML file pointing to the foobar js scripts. Whenever you edit something within foobar scripts, it gets automatically loaded on the html. And that's the first image, you can zoom, move, hover the nodes and look for the IDs for debugging. With that tool I have managed to solve some problems I had at the graph today and added Folk and Classical clusters too, so that should make all popular music up to date (but the "World music" at musicmap web).

For that part, I would gladly accept some help rendering the map according to link weighting (i.e. the real precomputed distance) instead of the viva methods by link centrality. That goes beyond my pretty limited knowledge of js and would be the final piece.

Re: [SMP] Music Graph development

Reply #2
First my request: I'm looking for a few users to try and test a SMP collection of scrips I have been working on.

I wasted a bunch of time with MusicIP back when it was beta - a lot of potential, as it generated the best related mixes I ever encountered. However it had the slowest track scanning ever and proprietary model that rendered it quickly useless.

I'm interested in yours - count me in. How?

It sounds great and also like it could lead to some personally long wanted FB statistical functions that have been out of my depth so far.

Re: [SMP] Music Graph development

Reply #3
Oh thanks! Just check your pm inbox in a few days ;)

This one generates mixes as good or better than musicIp but you need to have variety on your library. i.e. if you only have 20 folk tracks, then "folky playlists" will be really limited... while musicIp didn't care and filled the gaps with anything until it filled the minimum playlist length. The good part is that anything is configurable here, so you can lower the score filtering too...

This script also can be set to just output similar style/genre tracks or by mood or whatever, so you can also emulate those musicIp remixes which had different genres but "natural" variations.

I would want to recreate dynamic variations too. MusicIp was able to use one track as reference, output some similar tracks, use one of them as new reference for the rest of the playlist and so on. Also I could add more control to the content of the playlist, like X 80% similar tracks, Y 50% similar tracks, etc.Currently it's totally random, by score order (only higher scored tracks get to playlists), or by score order with random probability of being picked (it starts at high scored tracks but there is some randomness). MusicIp had so many details that I could work on this during so much time.

I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!

I'm currently polishing a few things so I can export all the files to you or anyone. Also managed to put custom images or shapes as nodes on rendering.
X

I have to create some buttons too (since I'm currently using the scripts as  "File\SMP" menus) and I will be ready for betatesting.

Re: [SMP] Music Graph development

Reply #4
I feel like a lot of this could be simplifed/made redundant by just smart tagging and understanding how we tend to categorize things in our heads.

I defaulted to just tagging common elements in music instead of following a rigid tree branching structure that people tend to follow. Each artist just has a bunch of stickers that range from very broad to oddly specific. The specificity is increased in areas of your library where you have the most knowledge/gathered the most amount of songs. And the only purpose is to help you quickly gather any subset of artists that have something in common.

Example stickers, ranging from more broad to oddly specific:

Basic genre of the song:
- This is always filled out as your starting point
- This is just to get the most basic idea of how the music sounds like.
- Lots of artists don't fit just one genre, so I often tag them with 1-3.

Subgenres equivalent:
- This is for areas of your library that has grown significantly and you need to break them down into smaller groups
- This requires that you actually learned to spot the small nuances of a genre and you actually care about the distictions
- If all "Electronic" music sounds the same to you, there's no point in meticulously tagging its subgenres
- But if "Melodic" or "Gothic" Metal has meaning to you, those subset of artists are worth tagging more precisely
- More generic terms tend to work better. E.g. "Melodic" can refer to Metal, or Rock, or even Classical music, or Industrial can refer to Rock, Metal or even Electronic music (if you approach it in a specific way).
- This way you tend to get a lot of cross-pollination between genres when building playlists

Moods/themes/feelings:
- This is fairly personal choice. It's more like a twist or "angle of attack" to further differentiate tracks within the same genre and subgenre.
- Words like "Modern", "Bleak", "Upbeat", "Downtempo/minimal" come to my mind. These are similar to subgenres but I'd say more abstract.
- They are more easily repeated across a wider arrays of genres/subgenres that don't otherwise have much to do with each other (you can have an upbeat song from any Electronic, Rock, Pop or even Metal song etc.)

Instruments used/location specific
- If you want you can specifically mark artists or song that emphasize an instrument or lack of
- For example music with standout drum usage (regardless of whether it's EDM, Classical or Metal music)
- Territory specific music that tend to use a similar set of instruments/style of music (E.g. Arabic or Asian)
- Instrumental songs vs songs with Female or Male vocal lead

Easy listening:
- This is to simply mark songs that are specifically easy, or hard to listen to, to filter potentially problematic songs from your autoplaylists
- Tracks don't tagged are counted as neutral

With this kind of tagging you can quickly get different kind of sets of music (either very broad, or very specific), or use them to generate nice autoplaylists with not that many rules in the query:
- All electronic music
- Only Melodic Metal music
- All gothic sounding music (ranging from Pop, to Metal, to Rock, to even Classical music)
- Electronic or Rock music that has no vocals but is generally upbeat
- Film scores that sound "Modern" and has a heavy emphasis on drums
- Anything that has an "Industrial" sound to it
- Japanese Pop music with female vocals
- etc.



Apart from these I also tend to use ratings in two ways:

Ratings supplied by the listener:
- People are not very good at rating objectively in a capped system (e.g. people usually rate from 7 to 10 in a 10 based system, unless the track is completely terrible)
- People's taste changes over time so what was rated 5/5 might not be subjectively thought as a 5/5 later
- In my system you can press a button to increase the songs rating by 1.
- Any song that's 1 or above is considered a favorite and can be included in autoplaylists
- Songs that are rated below 0 are banned and won't be part of normal listening
- Ratings don't have an upper cap so it's a numbers race. After sufficient amount of time passed you get a few tracks that are rated absurdly high and you can make queries based on this to filter out the "best" songs

Ratings tracked by the computer and the listener can't tamper with:
- Using the DAR system for this
- People tend to cheat with the ratings and artifically inflate them, if they get the choice (such as pressing the button multiple times)
- This represents the real time listened vs the subjective rating
- Using both numbers you can reasonably estimate the actual top songs
- Or just gradually phase out songs that have been listened to a lot, or force yourself to listen to those that got not much playback time



With the combination of the two things above you can make really well personalized sets of tracks that fit your current mood really well (via Facets or CUI Filter Panels).

My only gripe is that the random pools component (which has an option to basically normalize how often an artist's songs are presented) doesn't really let you save different presets of finely-tuned playlist generation rules, or that foobar can't really keep a running tally of "this artist has been played before in the last 20 minutes", to give equal chance to artists who only have a couple tracks.

Re: [SMP] Music Graph development

Reply #5
Btw it's already done and working on my side so there is no need to simplify things; I'm just looking for people to test it.

Your approach is the base of the things I'm discussing, but not the end. Yes, you can manually create playlists if you have tags set like that. That's what I have done on my library and it's what it's required for the scripts to work!

But that's not the point. I have my library tagged exactly like your description and I miss tons of things. Specially creating playlists dynamically. If I told you to select tracks similar in mood to the current one, you could spend 10 minutes creating combinations of moods...What is similar? 1 mood? 2? 5? Then the same with genres, with styles, etc. You know British Folk is similar to Folk Baroque, but .. that's in your head. Foobar knows nothing about "genres". You have to put that in tags in some way.

Autoplaylists, facets and all that have many weak points:
-Require direct user interaction. Queries become unmanageable for complex things. The user must check every playlist created to be sure they are right (duplications, ratings, etc.).
-They can not be changed dynamically. Yep, you can create a query for rock with moods "acoustic, sad, agressive, blabla", But then if you want the same for Folk, from specific dates you have to recreated the entire thing manually
-It's impossible for an human being to translate combinations of tags into queries manually. Check this image with a 15 lines query: https://hydrogenaud.io/index.php?action=dlattach;attach=18631;image
-They can not be set according to a source track. Yep, you can look for Rock or Glam Rock and all that. But you can not create an autoplaylist or Facets filter with "x same tags" to the current selected track.
-Can not be saved. Facets solves that partially.
-Can not filter tracks. You apply a query and that's all. Forget about filtering duplicates or other similar things. In fact, queries are created -by definition- to give you all the tracks found. Duplicate detection is out of the equation. I have many versions of the same songs, I don't want to listen to an autoplaylist with the same track 5 times.  "Top 25 tracks" playlist is a clear example of this problem.
-Exclusions are always manually set: Heavy Metal but not Black Metal and Folk Metal is OK but Folk Metal and Folk-Rock is not, and...
-Can not set playlist lenght.
-Require a crazy amount of work to make it work properly. Yes, you can add more and more and more tags for corner cases and complicate queries... but it makes no sense at some point. I want a system which identifies folk sub-genres and their relations, not having to add a trillion of tags to differentiate them (Instruments used/location specific & Easy listening:); work is done once, etc.
-What about the day you just want to listen to music and not to spend time with all those silly things?

Essentially it's simple. Foobar already has all the ingredients to do things like modern music players, but ironically they have not been mixed yet. Random pools is an example of that, it's development is against that aim of fine-tuning playlists. For ex. it has already been asked to assign pools to playlists names, but that request has been rejected due to being "too complicated".

Even foobar, which already has many title-format variables, omits the current selected track or playing track data. If foobar gives us %currentplaying_genre" or things like that, it would be a joy to create dynamic auto-playlists or random pools according to the selected track. You can not create playlists using other playlist as source, etc. Check MusicBee for those details missing.

We are at a point where machine learning, automatic tagging or music recognition are "simple" and userfriendly, so what I expect about a modern player is applying those concepts locally. I don't like Spotify because it only works via internet. Do you remember Genius from itunes? The same. I want the same things found on those players but working on my PC. I don't want to learn 8000 queries and tags to "teach" foobar to make playlists, I want foobar to show me my local music the same other players do... finding related things automatically. And in the process, to offer many more things.

Re: [SMP] Music Graph development

Reply #6
Just finished making buttons to test it easily. Also for the other functions, with hover text, and ui changing according to preferences.
XX
XX
Quote
For that part, I would gladly accept some help rendering the map according to link weighting (i.e. the real precomputed distance) instead of the viva methods by link centrality. That goes beyond my pretty limited knowledge of js and would be the final piece.
And somehow I managed to solve that.
X

Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
But that still applies.

Sent you a PM mjm716

Re: [SMP] Music Graph development

Reply #7
Development continues and I have added some new things to the set of scripts.

-Buttons framework to merge/add/create new buttons seamlessly to any SMP bar. That means... you create a button file for whatever you want (sort the playlist for example) and then you simply "include it" on a main file. The main buttons file loads all included buttons, sets position, etc. Zero coding (and no need to edit the other files to merge them). Enabling or disabling buttons just takes 1 line.
X

-Included some example buttons. An user configurable button within foobar (using properties panel) and 2 new buttons with new functionality (some auto-tagging processing and playlist filtering).

-Updated the HTML rendering file with a few things.... it precomputes the graph before rendering it, pins the node info, etc. Font and label sizes are now set according to window size.
X
X

-Added extensive comments over the scripts.

-Added a remove duplicates function which can be used as playlist filtering, setting how many duplicates are allowed... you can filter by date and only allow 3 tracks per date or things like that. And speed improvements (thanks to snotlicker)

Betatesting still required! Thanks :)

Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
And help required for that yet!

Re: [SMP] Music Graph development

Reply #8
Wow! That's really impressive!
I could not even imagine that such cool stuff would be performed via my component!
Keep it up! ;)

Re: [SMP] Music Graph development

Reply #9
Quote
I want to implement fingerprinting comparison too at some point. Using either AcousticID fingerprints or libfooid. To find duplicates, similar track versions, ... but that's currently delayed since my js knowledge is pretty limited right now hahaha If someone points me to JS code to de-structure the fp I would be so grateful!
And help required for that yet!

You would need raw output of fpcalc for the files.

These links may help !

https://groups.google.com/g/acoustid/c/C3EHIkZVpZI?pli=1

https://gist.github.com/hertzg/c75aa10072d10f00b38e

Re: [SMP] Music Graph development

Reply #10
Wow! That's really impressive!
I could not even imagine that such cool stuff would be performed via my component!
Keep it up! ;)
Thanks! Well the html rendering part is done outside foobar with a browser, so it's just using standard js code. I'm not sure if it could be integrated in any way within foobar as a html popup though (?) On github, you told me IE is used for html rendering, so I could try that path once I install it. I don't use fancy things, just standard js, SVG rendering and jquery as a library. It would be awesome: I could do things like showing tracks from the library matching the node selected, playlists by node and "radius", etc.

The graph creation is done with a node.js library, but works within foobar and browsers; I simply had to change a few things on the ngraph library to make it work.

The distance calcs and playlist and all that is done within foobar, your component is fantastic I think any functionality from modern web players can be replicated locally without problems with js. This is just an example. The only limitation I have found is processing time. Spotify has tons of servers calculating these same things! It takes some seconds on our PCs to create a graph or distance calculations for all the library. I even think machine learning models can be paired with your component for auto-tagging or things like that. We just have to see js as a door which allows us to load arbitrary things within foobar and manipulate the library and the playlist with all that info!

Quote
You would need raw output of fpcalc for the files.

These links may help !

https://groups.google.com/g/acoustid/c/C3EHIkZVpZI?pli=1

https://gist.github.com/hertzg/c75aa10072d10f00b38e

Let's see if I find a way to calc the raw fp within foobar (so users don't have to analyze things with other programs*) and the comparison logic. Thanks! I will take a look at that.

*Foobar has a component but uses other fp method. Ideally, I would support both.

Re: [SMP] Music Graph development

Reply #11
Some advances:

-Managed to create buttons bar as easy as just include the button file. 1 line.

-Buttons bar can be merged too -> You just include the button bar path, and it's merged the same than standalone buttons.

-Buttons and bars can be duplicated just by including them twice. This is important since they act like different instances, with their own properties at the panel (they get duplicated with prefixes), variables and all that! (other implementations required iDs to be different, properties description adjusted before merging, creating new files for "copies", etc.)

-All these things mixed means... you can just copy the "include" line of a button example 10 times and configure each button within foobar (properties panel) as you wish. In other words, you can create customized buttons using a predefined template without coding a line. You only need to copy/paste paths at the js file and the rest is adjusted on the fly.

X

X

X

X
In this example, every button does the "search by distance" thing explained on the thread according to different user set variables. That means I don't have to create a predefined set of buttons which creates playlists according to my liking... but the user can create their own in a few seconds.

The great part of the framework is that it's universal. It can be applied to anything, not just my intended use.

-"Similar by" arbitrary tags / number of coincidences. If date is used, then it checks by range instead of number of coincidences. It's a generalization of "search same" functions, user can put whichever tag they want and number of coincidences required for the query. Using it, creates a dynamic query according to the current selected track, and creates a playlist without duplicates (using my remove duplicates function) with random entries from that query. It would be a query equivalent to the "search by distance" implementation. Obviously the query is limited to "equal tags" and simple things while the other uses a graph to find relations between genres/styles and more complex things...

X
Tooltip is adjusted according to the panel properties and tags set.
X
Query example created.
X
Properties panel.





Re: [SMP] Music Graph development

Reply #12
-Remapping and merging tags feature for similarity checks. So not only you can use your own tag names, but also merge different fields into one before evaluating them. This allows us to use the scripts with any tag scheme without needing to re-tag.
"genre tag" -> genre, allmusic_genre, ...
X

-Added 2 custom tags for checking. There is specific logic for genre, style, date, etc. but you can also add 2 custom tags to be checked by matching (strings) or within a range (numbers) according to their own weights. Why? Let's say some user have it's own "occasion" tag, then it can be used on the search by similarity buttons/functions without changing the code. It's just a placeholder for user set tags, appart from the main ones.
X

-Some speed improvements (even with the tag merge/remap feature added).

-Filter for the final playlist. Apart from removing duplicates (by title-date-artist, not just duplicated files),  also a filter can be set to only allow X tracks per artist, per date, etc. Fully configurable (tags and number of allowed tracks per check).

Work is mostly done on this thing, I'm now working on a playlist manager which doesn't force you to have playlist loaded within foobar, so you only load whatever you want. (like a library viewer / album list for playlists).

Right now if you have 100 playlist, you must have them loaded on the program, cluttering the playlist tabs... well it makes no sense to me. Other playlist managers alleviate the issue, but you still need to have them all loaded and then "explore" the playlists list using the manager instead of the tabs. Tabs become totally useless and unmanageable:
X

My approach is different, you have your list of playlist and you simply load the ones you want as new tabs. That way you have the best of both worlds: playlist tab functionality (without cluttering) and a playlist manager. There is something similar for autoplaylists (by saving queries) on SMP, but it's limited to that: auto-playlists.

Re: [SMP] Music Graph development

Reply #13
Playlist manager is ready to be tested too.  Please drop me a line here: regorxxx@protonmail.com
Note it currently uses a rewritten version of the list panel by marc2003, so the UI is pretty similar right now. I plan to change that once I see the internals are working.

A copy of my notes:
Quote
/*    Playlist Manager v 0.1 02/02/21
   Manager for offline playlists. Shows a virtual list of all playlists files within a configured folder (PlaylistPath).
   Features:
      - Auto-saves changes within foobar to playlists files. (configurable)
      - Automatically updates changes within the tracked folder. (configurable)
      - Lock/unlock playlists (so they are read-only).
      - Automatically locking of native foobar playlists files (.fpl). (configurable)
      - Playlist unique IDs. You can have multiple playlists with same name on the UI and binded to different files. (configurable)
      - Show playlist size on the list. (some limits apply for .fpl playlist files) (configurable)
      - 3 different writable formats. (some limits apply for .pls playlist files) (configurable)
      - RecycleBin: deleting and restoring.
   TODO:
      - Merge with autoplaylists ?
      - Search box
      - List ordering by: Name / size / Virtual folder ("category")
      - Add category data to playlists for searching and ordering
      - Drag n drop to add playlists to folder
      - Better fpl support?
         * Save data to json, load data from json, overwrite only non fpl files
         * Replace 'update playlist' with save file UI for fpl files, when locked
      - Replace 'update playlist' for locked files with popup to overwrite file but maintaining locked status
         
*/
Resume: you have a folder with 20 playlists and these are NOT shown on foobar tabs. You only load the one you want and that's all.
You can treat playlists as files, you can delete them and all that.
If you have auto-update enabled, the folder is tracked... so if you add or remove playlists using the explorer the changes reflect into foobar immediately (the same than album list).
If you have auto-save enabled, then changes to loaded playlists are monitored. Whenever you add tracks, change order, etc. the playlist file will reflect the changes. You can also have locked files (so they can only be updated manually).

Red playlists are locked
X

Right and left buttons menus
X

The playlist format: currently supports m3u8 and m3u (full features), pls (no unique Ids, no locked status) and fpl (only read).
X

I managed to create unique IDs with invisible unicode chars. The (*) is just for identifying them for demonstration purposes. That means you can have playlist with duplicates names on the UI and still be linked to their unique files.
X

And finally the point is these are real files and folders (not saved on a universal json file), so you can have multiple instances of the panel pointing to different sets of playlists:
X

Re: [SMP] Music Graph development

Reply #14
This is what I was talking about. The panel title is the folder name.
X
And every folder has its own playlists.
X

It could work as categories with different sub-folders for every one (although I plan to show the folders as a tree at some point).
Finally, the files are ready to be used in any other program. That's the main difference between a json approach and this one. The files are not limited to foobar internal use (neither require additional code).

This is wip, but tooltip shows current foobar playlist name associated, number of tracks, file name, if the playlist is locked for modifications, category and possible tags given to the playlist (for future searching feature)....
X

When I merge it with autplaylists, then the tooltip will display other data, like query (up to some length),  number of tracks which will be generated, etc.

Re: [SMP] Music Graph development

Reply #15
Some updates:
- Different Icons for  empty/non empty playlists. Along the red color for locked playlists.
- Double click on a playlists automatically opens it. If it's already open, it makes it the active playlist
- Double clicking works along single clicking and tooltip.
- Added selection indicator.
- Added playlist count to the header.
- Category and tags fully implemented (it's saved on the playlist files too). Next step is a search box or a dynamic menu generated by those categories/tags.
- Added buttons to sort: the button names, tooltip and functionality  changes when clicking. So there are 2 sort modes and 3 filter modes which you can cycle.

X

X

The biggest improvement is the buttons framework, since I have managed to just reuse the one I created for the buttons bar and make it compatible within arbitrary panels. i.e. buttons can be added to anything without complicating the code just by copying a few lines and reusing the same files. It's portable. My biggest gripe when looking to other UI samples and complete themes is they are heavily dependent on tons of functions and variables to the point is a nightmare to reuse them.

I would want to do the same with a scroll bar and a search box, but not sure how many time it will take.

It's fully functional so if you want to test it please drop me a line here: regorxxx@protonmail.com

Re: [SMP] Music Graph development

Reply #16
More updates:
- Added autoplaylists to the manager. They can be created, modified, deleted and saved the same than standard playlists.
- Standard playlists are linked to physical files on the tracked folder.
- Autoplaylists are saved to a json file. i.e. every-time the panel is updated, they are merged with the list of physical files.
- Tooltip shows the query and sort for autoplaylists.
- Tooltip shows the number of tracks the query would output (always updated to the current foobar database).
- fpl playlists have their size cached when you load them for the first time on foobar. That size is cached (so it shows in the UI even if there is no way to "read" fpl playlists).
- Added a second filter to show/hide autoplaylists. (thinking about a 3 cyclic filter: only playlists, only autoplaylists, both)
- Added a different color to autoplaylists.

At the current state it contains every functionality of Auto-playlist Manager by marc2003 for autoplaylists (and some more).  Since I was using that panel too before creating this (and I have tons of lists there), I could add some helpers to migrate the json autoplaylist file to this manager. And it would be helpful for some people too.

Tooltip for AutoPlaylists:
X

Contextual menu for AutoPlaylists:
X

Contextual menu for standard playlists:
X

Panel menu:
X

Re: [SMP] Music Graph development

Reply #17
More updates:

X
- Added a menu entry to import json files created by Auto-playlist Manager by marc2003. Migrating autoplaylist from that panel to this one just takes a matter of seconds.

X
- AutoPlaylist size (*) is updated on json import (so they show the same info than autoplaylist created within this panel).
- AutoPlaylist size is calculated only once at script loading. Can be forced to refresh with 'manual refresh'. This is done to reduce lag, since 1x autoplaylist can easily take a few seconds to compute.
- Current index (selected item) and offset (position within the list when scrolling down) is not reset when updating the panel, applying filter, etc.

(*) The number of tracks the query would output.

Re: [SMP] Music Graph development

Reply #18
I have received some mails, I will answer you all asap with new links as soon as I clean a bit the graph script files I did after these changes:
https://hydrogenaud.io/index.php?topic=120394.msg992987#msg992987
Just to be sure I give everybody the last working version.

Back to the playlist manager, More updates:
-I focused on configuration, some safety checks and coding it in a way it can be easily modified (for adding new filters/sorting methods, unique IDs, etc.). And the properties panel and menus are recreated on script init automatically according to those methods (while saving the previous user variables). Should be easy for anyone with basic knowledge to add their own things there.

-Properties descriptions change according to things set on the panel, not just the values. i.e. if you change the sort method using the contextual menu, then the description on the panel reflects the associated states dynamically.
X

-Sort button can be changed to sort by Name, Category or Size. UI updated accordingly.
X

-Sort state is saved between different sessions/reloads.
-Sort method is saved between different sessions/reloads.
-Filters are saved between different sessions/reloads (configurable).
-Added contextual menu options for:
   -Sort method: sort by Name, Category or Size
   -Show Playlist Size: All (refresh autoplaylists queries), only standard playlist (1), no.
   -UUIDs: invisible unicode chars plus (*), (a-f), (*) or nothing.
   -Filter saving
-All playlist are loaded in cache once, filtering just changes the "painted" playlist on the list. This fixes autoupdate firing when using filters (and other complicate workarounds). Speed improvements.
-If you choose not to refresh autoplaylist sizes (1), then the original size gets used. Original size is always calculated when importen from json or creating the autoplaylist (done once in ther entire lifetime). Greatly speed ups panel loading while maintains some size usability. They can always be refreshed manually though.
-Json file backups periodically on script init/reload. Old file gets copied to .old, so there is always a previous version to restore if something gets broken.
-Modified button framework to allow on the fly coordinates calculation, so it merges the recalc of buttons bar with width and height windows resizing.

 

Re: [SMP] Music Graph development

Reply #19
More updates: (last ones before I send the new batch of links)
- New updates are delayed when performing internal updates/processing to avoid conflicts when editing files.
- Separators between different names/categories (configurable). Numbers are grouped by #
- Now playing and loaded playlist indicators.
- Selection rectangle indicator.
X

- Background text is wrapped when text width is > than panel width. When showing playlists, the playlist name is trimmed.
X

Code: [Select]
	TODO (by relevance):
- Better fpl support?
* Save data to json, load data from json, overwrite only non fpl files
* Replace 'update playlist' with save file UI for fpl files, when locked
* Native FPL support ? (requires new SMP version)
- UUID:
* nextId() called only for new playlist, reuse old id?
* Reassign UUIDs when changing method?
+ First save loaded playlist and then close them?
+ Rename loaded playlists?
* Save UUID on json but not on files?
- Search box
* By Name
* By tag
* By category
- Drag n drop: (requires new SMP version)
* Add playlists files to folder (requires new SMP version)
* Add tracks to playlist file (requires new SMP version)
- Scrollbar

Re: [SMP] Music Graph development

Reply #20
Added configurable colors for different elements of the playlist manager and fixed some bugs for the GRAPH files. The main difference is the ability to remap tags and use TF expressions which should cover everyone's use-case.
https://hydrogenaud.io/index.php?topic=120394.msg992987#msg992987

Just sent a mail with a new link to everyone who asked for it.

Next things to do for the graph files is considering adding some configurable tag filtering (like filtering 'soundtrack' when comparing genre values), adding more genre/styles and links to the map and maybe some more complex link logic to account for influences (reducing the total mean distance by an absolute value).

Would love to offer better fpl support (write/reading/updating without user intervention) on the playlist manager, but since it's a closed format and there are no funcs exposed to work with them I can not do anything more. Sorry.

Re: [SMP] Music Graph development

Reply #21
More updates for the Graph scripts:
- Configurable tag filtering (like filtering 'soundtrack' when comparing genre values). So those tags are not considered for scoring purpose.
- New linking logic to account for influences when they happen between 2 nodes within the same SuperGenre (reducing the total mean distance by an absolute value the opposite to how 'anti-influence' works).
- Fixed a small bug on linking between Industrial_cluster, Four-to-the-floor Dance_cluster and Downtempo_cluster when linking in circles.
- Added extensive debug checks when loading graph scripts within foobar or HTML, with popup message and console warnings if there are errors on the graph descriptors:
      +Not connected nodes.
      +Nodes connected to their-selves.
      +Substitutions with influence values != 0.
      +All distance variables to create the graph (according to some logic).
      +Basic path testing (SuperGenres, SuperGenre Clusters and SuperClusters in circles), distance should be always greater than a threshold (according to distance variables) and be a finite value.

The debug checks should cover most common errors when editing the graph descriptors.

Re: [SMP] Music Graph development

Reply #22
More updates on the graph:

- Some speed improvements (below 3 secs for 40 K tracks on i7 920 from 2008).
- Added extensive documentation for its use and aim, check below.

Code: [Select]
/*	
Search by Distance v 0.2.1 17/02/21
-----------------------------------
Creates a playlist with similar tracks to the currently selected one according to genre, style, key, etc.
Every track of the library is given a score according to those tags and/or a distance according to genre/style.
When their score is over 'scoreFilter', then they are included in the final pool.
After all tracks have been evaluated and the final pool is complete, some of them are chosen to populate
the playlist. You can choose whether this final selection is done according to score, randomly chosen, etc.
All settings are configurable on the properties panel (or set in the files when called using buttons, etc.)
Check the descriptions of the properties panel to check how the variables work.

These are the weight/tags pairs checked by default:
- genreWeight : genre - styleWeight : style
- dyngenreWeight : virtual tag - moodWeight : mood
- keyWeight : key - dateWeight : $year(%date%)
- bpmWeight  : bpm - composerWeight : composer (unused by default)
There are 2 custom tags which can be set by the user too:
- customStrWeight : (unused by default) - customNumWeight : (unused by default)

Any Weight/tags pair can be remapped and/or merged (sep. by comma).
For example, linking genreWeight to 2 different genre tags on your files:
- genreTag : allmusic_genre,my_genre_tag

Some weight/tags pairs can be linked to TitleFormat Expr. Use tag names instead of TF expressions when possible (+ performance).
For example, see dateWeight: TF is used to have the same results for tracks with YYYY-MM tags or YYYY tags.
- dateTag : $year(%date%) -customNumWeight : (unused by default)

Genre and Style tags (or their remapped values) can be globally filtered. See 'genreStyleFilter'. Case sensitive.
For example, when comparing genre values from track A to track B, 'Soundtrack' and 'Radio Program' values are omitted:
- genreStyleFilter: Soundtrack,Radio Program

There are 3 methods to calc similarity: WEIGHT, GRAPH and DYNGENRE.
- WEIGHT: -> Score
Uses genreWeight, styleWeight, moodWeight, keyWeight, dateWeight, dateRange, bpmWeight, bpmRange, composerWeight, customStrWeight, customNumWeight + scoreFilter
Calculates similarity by scoring according to the tags. Similarity is calculated by simple string matching ('Rock' != 'Soul') and ranges for numeric tags. This means some coherence in tags is needed to make it work, and the script only works with high level data (tags) which should have
been added to files previously using manual or automatic methods (like MusicBrainz Picard, see note at bottom).
- GRAPH: -> Score + Distance
Same than WEIGHT + max_graph_distance
Apart from scoring, it compares the genre/styles tags set to the ones of the reference track using a graph and calculating their min. mean distance.
Minimum mean distance is done considering (A,B,D) set matches (A,B,C) at 2 points (0 distance). So we only need to find the nearest point
from (A,B,D) to (C) which will be x. Then we calculate the mean distance dividing by the number of points of the reference track : (0 + 0 + x)/3
Imagine Google maps for genre/styles, and looking for the distance from Rock to Jazz for ex. 'max_graph_distance' sets the max distance allowed, so every track with genre/styles farther than that value will not be included in the final pool. Note this is totally different to simple string matching, so 'Acid Rock' may be similar to 'Psychedelic Rock' even if they are totally different tag values (or strings).
This method is pretty computational intensive, we are drawing a map with all known genres/styles and calculating the shortest path between the reference track and all the tracks from the library (after some basic filtering). Somewhere between 2 and 5 secs for 40 K tracks.
For a complete description of how it works check: 'helpers/music_graph_descriptors_xxx.js'
And to see the map rendered in your browser like a map check: 'Draw Graph.html'
- DYNGENRE: -> Score
Same than WEIGHT + dyngenreWeight
Uses a simplification of the GRAPH method. Let's say we assign a number to every "big" cluster of points on the music graph, then we can simply
put any genre/style point into any of those clusters and give them a value. So 'Rock' is linked to 3, the same than 'Roots Rock' or 'Rock & Roll'.
It's a more complex method than WEIGHT, but less than GRAPH, which allows cross-linking between different genres breaking from string matching.
For a complete description of how it works check: 'helpers/dyngenre_map_xxx.js'

Arguments: (genreWeight, styleWeight, dyngenreWeight, moodWeight, keyWeight, dateWeight, dateRange, bpmWeight, bpmRange, playlistLength, probPick, scoreFilter, max_graph_distance, bRandomPick, method, forcedQuery)

A: do_searchby_distance(15, 10, 0, 5, 10, 25, 15, 5, 25, undefined, 100, 60, 150, 1, "GRAPH", undefined)
// Random mix with only nearest tracks, most from the same decade

B: do_searchby_distance(10, 10, 0, 5, 5, 25, 15, 5, 25, undefined, 100, 60, 250, 1, "GRAPH", undefined)
// Random mix a bit varied on styles (but similar genre), most from the same decade

C: do_searchby_distance(0, 5, 0, 15, 10, 25, 15, 5, 25, undefined, 100, 50, 600, 1, "GRAPH", undefined)
// Random mix even more varied on styles/genres, most from the same decade

D: do_searchby_distance(0, 5, 0, 15, 10, 0, undefined, 5, 25, undefined, 100, 50, 900, 1, "GRAPH", undefined)
// Random mix with different genres but same mood from any date

Note about genre/styles:
GRAPH method doesn't care whether "Rock" is a genre or a style but the scoring part does! Both values are considered points without any distinction.
Genre weight is related to genres, style weight is related to styles.... But there is a workaround, let's say you only use genre tags (and put all values
together there). Then set style weight to zero. It will just check genre tags and the graph part will work the same anyway.

Note about GRAPH/DYNGENRE exclusions:
Apart from the global filter (which applies to genre/style string matching for scoring purpose), there is another filtering done when mapping
genres/styles to the graph or their associated static values. See 'map_distance_exclusions' at 'helpers/music_graph_descriptors_xxx.js'.
It includes those genre/style tags which are not related to an specific musical genre. For ex. "Acoustic" which could be applied to any genre.
They are filtered because they have no representation on the graph, not being a real genre/style but a musical characteristic of any musical composition.
Therefore, they are useful for similarity scoring purposes but not for the graph. That's why we don't use the global filter for them.
This second filtering stage is not really needed, but it greatly speedups the calculations if you have tons of files with these tags!
In other words, any tag not included in 'helpers/music_graph_descriptors_xxx.js' as part of the graph will be omitted for distance calcs,
but you save time if you add it manually to the exclusions (otherwise the entire graph will be visited trying to find a match).

Note about GRAPH substitutions:
The graph created follows a genre/style convention found at 'helpers/music_graph_descriptors_xxx.js'. That means than only things written there, with exactly
the same name (including casing) will be found at the graph. As already noted, 'rock' will not be the same than 'Rock', neither 'Prog. Rock' and 'Progressive Rock'. This is a design decision, to force users to use a tag convention (whatever they want) and only one. See last note at bottom. As a workaround, since you
don't have to follow my convention, there is a section named 'style_substitutions' which lets you tell the scripts that 'Prog. Rock' is the same than
'Progressive Rock' for example. So once you have all your library tagged as you please, you can either mass replace the different terms to follow the convention
of the graph OR add substitutions for all of them. It has other complex uses too, but that goes beyond this doc. Check 'helpers/music_graph_descriptors_xxx.js'.

Note about console logs:
The function assigns a score to every track and you can see the names - score (- graph distance) at console. The selected track will get a score = 100 and distance = 0. Then, the other tracks will get (lower or eq.) score and (greater or eq.) distance according to their similarity. You can toggle logging and profiling on/off setting the booleans within the function code (at their start).

Note about graph configuration and error checking:
By default 'bGraphDebug' is set to true. That means it checks the graph files whenever you load the script (usually once per session). You can disable it (faster loading). If you edit 'helpers/music_graph_descriptors_xxx.js' file, it's advised to enable it at least once so you can check there are no obvious
errors. The html rendering helps at that too (and also has debugging enabled by default).

Note about buttons framework:
When used along buttons framework, you must pass all arguments, along the new prefix and merged properties! See "buttons_search_bydistance_customizable.js"

Note about High Level data, tag coherence, automatic tagging and MusicBrainz Picard:
Network players and servers like Spotify, Itunes Genius, YouTube, etc. offer similar services to these scripts. Whenever you play a track within their players,
similar tracks are offered on a regular basis. All the work is done on the servers, so it seems to be magic for the user. There are 2 important caveats for
this approach: It only works because the tracks have been previously analyzed ('tagged') on their server. And all the data is closed source and network
dependent. i.e. you can always listen to Spotify and your great playlist, at least while you pay them, those tracks are not removed for your region and you
have a constant Internet connection.

That music listening model has clear drawbacks, and while the purpose of this caveat is not talking about them, at least we have to note the closed source
nature of that analysis data. Spotify's data is not the same than Youtube's data... and for sure, you can not make use of that data for your library in any way.
Ironically, being at a point where machine learning and other methods of analysis are ubiquitous, they are mostly relegated behind a pay-wall. And every time
a company or a developer wants to create similar features, they must start from scratch and create their own data models.

An offline similar program which does the same would be MusicIP ( https://spicefly.com/article.php?page=what-is-musicip ). It appeared as a viable alternative
to that model, offering both an Internet server and a complete ability to analyze files offline as fall-back. Nowadays, the company is gone, the software
is obsolete (although usable!) and documentation is missing for advanced features. The main problems? It was meant as a standalone solution, so there is no
easy way to connect other programs to its local server to create playlists on demand. It can be done, but requires manually refreshing and maintaining
the server database with new tag changes, data analysis, and translating ratings (from foobar for ex.) to the program. The other big problem is analysis time.
It may well take a minute per track when calculating all the data needed... and the data is also closed source (so it has no meaning outside the program).
The reason it takes so much time is simple, the program was created when machine learning was not a reality. MusicIP may well have been ahead of its time.

Back to topic, both online and offline methods due to its closed source nature greatly difficult interoperability between different programs and use-cases.
These scripts offer a solution to both problems, relying only in offline data (your tags) and open source data (your tags again). But to make it work,
the data (your tags) need relevant info. Since every user fills their tags without following an universal convention (most times leaving some tags unfilled),
the only requirement for these scripts to work is tag coherence:
- Tags must point to high-level data, so analysis (whether computational or human) must be done previously. 'Acoustic' or 'Happy' moods are high level data,
'barkbands' with a list of values is low-level data (meant to be used to calculate high-level ones). See this for more info: https://acousticbrainz.org/data
- Tags not present are simply skipped. The script doesn't expect any specific tag to work, except the obvious ones (can not use GRAPH method without
genre/style tags). This is done to not enforce an specific set of tags, only the ones you need / use.
- Tags are not hard-linked to an specific tag-name convention. Genre may have as tag name 'genre', 'my_genre' or 'hjk'.
- Basic filtering features to solve some corner cases (for genre/styles).
- Casing must be the same along all tags. i.e. 'Rock' always spelled as 'Rock'. Yes, it could be omitted using .toLowerCase(), but is a design decision, since it forces users to check the casing of their entire set of tags (ensuring a bit more consistency in other fields).
- Reproducibility all along the library set. And this is the main point of tag coherence.

About Reproducibility: If two users tag a track with different genres or moods, then the results will be totally different. But that's not a problem as long as
every user applies its own 'logic' to their entire library. i.e. if you have half of your library tagged right and the other half with missing tags,
some wrongly set and others 'as is' when you got the files... then there is no coherence at all in your set of tracks nor your tags. Some follow a convention
and others follow another convention. To help with that here are 2 advises: tag your tracks properly (I don't care about specific use-cases to solve what ifs)
and take a look at MusicBrainz Picard (that's the open source part): https://picard.musicbrainz.org/
Now, you DON'T need to change all your tags or your entire library. But Picard offers 3 tags using the high-level open source data of AcousticBraiz:
mood, key and BPM. That means, you can use your manual set tags along those automatically set Picard's tags to fulfill both: your tag convention and reproducibility along your entire library. Also you can manually fix or change later any mood with your preferred editor or player. Picard also offers plugins
to fill other tags like genres, composers, etc. Filling only empty tags, or adding them to the existing ones, replacing them, ... whatever you want.
There are probably other solutions like fetching data from AllMusic (moods), lastFm (genres), Discogs (composers), etc. Use whatever you want as long as tag
coherence and reproducibility are ensured.

What happens if you don't want to (re)tag your files with moods, bpm, key, splitting genres/styles, ...? Then you will miss that part, that's all. But it works.
What about a library properly tagged but using 'rock' instead of 'Rock' or 'African Folk' instead of 'Nubian Folk' (although that's a bit racist you know ;)? Then use substitutions at 'helpers/music_graph_descriptors_xxx.js' (not touching your files) and/or use mass tagging utilities to replace values (touching them).
And what about having some tracks of my library properly tagged and not others? Then... garbage in, garbage out. Your mileage may vary.
*/
/*
  Last changes:
- Speed improvements replacing arrays with sets in multiple places.
- Speed improvements on tag filtering.

TODO:
- Pass object as argument
- Overwrite substitutions with user set file:
- Put this after main file
include(fb.ProfilePath + 'scripts\\SMP\\xxx-scripts\\helpers\\music_graph_descriptors_xxx_user.js');
Merge old object property with the ones set there.
On new updates, only overwrite main file.
- Clean unused things
- Fingerprint comparison for scoring?
- Output just similar tracks by fingerprint
- Give more weighting to similar tracks
- Fine-tune Pre-filtering:
- Fine-tune queries
- GRAPH : Apart from individual link caching, cache entire distance from set (A,B,C) to (X,Y,Z)
- Different date formats or just TF? YYYY-MM-DD; YYYY-MM
- Make it easy to port the code to other frameworks outside foobar:
- Include are just replaced with import
- Properties are just objects
- All graph calcs and descriptors are independent of foobar so they work 'as is'. That's the main feature and heaviest part.
- All tag 'getValues' would be done using external libraries or native player's methods
- Queries would be disabled (they are not intrinsic part of the code, just a speed optimization)
- Handle lists should be replaced with similar objects
- Console logs work as they are.
- Helpers used don't need foobar at all (except the 'getValues' for tags obviously)
*/

Next I will add a short FAQ and how-to for adding new releases into AcousticBrainz, so anyone can get all data (mood,key, bpm) for their own files even if they were not in the server. Following a simple routine and using a Picard Plugin , it takes 3 minutes of user intervention.

Re: [SMP] Music Graph development

Reply #23
More updates on the graph:
- Added user file to overwrite\add\delete properties from 'music_graph_descriptors'. On new updates, it would never get overwritten... making it easy to maintain your own changes without having to manually merge files (or breaking the main file). If the value is added, deleted or replaced is done according to the logic described at 'music_graph_descriptors_xxx_user.js'.

Added this to the doc:
Code: [Select]
	Note about editing 'helpers/music_graph_descriptors_xxx.js' or user file:
Instead of editing the main file, you can add any edit to an user set file named 'helpers/music_graph_descriptors_xxx_user.js'. Check sample for more info.
It's irrelevant whether you add your changes to the original file or the user's one but note on future script updates the main file may be updated too.
That means you will need to manually merge the changes from the update with your own ones, if you want them. That's the only "problem" editing the main one.
Both the html and foobar scripts will use any setting on the user file (as if it were in the main file), so there is no other difference.
Anything at this doc which points to 'helpers/music_graph_descriptors_xxx.js' applies the same to 'helpers/music_graph_descriptors_xxx_user.js'.

Then a bit more info in the sample file:
Code: [Select]
/*
This is a sample file to show how to replace/add/delete properties at music_graph_descriptors without touching the original file.
Just copy properties from the original file 'helpers\music_graph_descriptors_xxx.js' to this one and edit whatever you want.
The replacement follow this logic:
-Always replaced: Strings , numbers, booleans
-Adds new values: new elements on arrays and sets.
-Replaces old values with new ones: elements present on both arrays
-Deletes old values: elements present on both sets
Note properties need a ',' after the values:
propertyA: value,
propertyB: 'valueB',
propertyC: [valueC, [subValuesC]],
...
*/

const music_graph_descriptors_user = {
/*
// You can either replace, delete or add array values to the original one
// This example would replace the original value with a new one.
style_supergenre: [
['Classical Modernist Era_supergenre', ['Third Stream']]
],

// This example would simply delete the old value (= replacing it with empty values)
style_supergenre: [
['Classical Modernist Era_supergenre', []]
],

// This example would add a new value
style_supergenre: [
['My Own_supergenre', ['My Own Style/Genre A', 'My Own Style/Genre B']]
],
*/

/*
// You can either delete or add set values to the original one
// Deletes 'Female Vocal' (since it was already present) and adds 'My New Set Value'
map_distance_exclusions: new Set([
'Female Vocal','My New Set Value'
]),
*/

/*
// You can replace numbers
primary_origin: 200,

// strings
nodeShape: 'circle',

// and boolean properties
bPreRender: false,
*/
};

The main point of this change is not only adding your own values... but also replacing or deleting default ones without touching the original graph descriptors. And doing it without complex edits, just doing the same than adding values.

It works both on html rendering and within foobar, so it's really easy to see if they work as intended just by refreshing the html file (for example adding new genres, changing distances, etc.). Since I added extensive debug functions on the previous updates, the user set values are also checked for consistency.

Re: [SMP] Music Graph development

Reply #24
More updates on the graph:
- Added extensive examples to use the script for different playlist mixes.
- Default arguments using object notation (easier to set only those you want).
- Pre-scoring filters
   - Anti-Influence filter: filter anti-influences by query for GRAPH Method.
   - Influence filter: gets only influences by query for GRAPH Method.
- Post-scoring filters
   - Allow only N+1 tracks per tag on the pool. Configurable. i.e. 2 tracks per artist, 4 tracks per genre, etc.
- Link cache pre-calculated on startup with styles/genres present on library (instead of doing it on function evaluation later).

Code: [Select]
/*	Arguments: 	As object, anything not set uses default values set at properties panel.
{ properties,
// --->Weights         
genreWeight, styleWeight, dyngenreWeight, moodWeight, keyWeight, dateWeight, bpmWeight, composerWeight, customStrWeight, customNumWeight,
// --->Ranges (for associated weighting)
dyngenreRange, dateRange, bpmRange, customNumRange,
// --->Pre-Scoring Filters
forcedQuery, // Query to filter library
bUseAntiInfluencesFilter, // Filter anti-influences by query, before any scoring/distance calc. Stricter than distance checking.
bUseInfluencesFilter, // Allow only influences by query, before any scoring/distance calc.
// --->Scoring Method
method, // GRAPH, WEIGHT, DYNGENRE
// --->Scoring filters
scoreFilter, sbd_max_graph_distance, // Minimum score & max distance to include track on pool
// --->Post-Scoring Filters
poolFilteringTag, poolFilteringN, bPoolFiltering, // Allows only N +1 tracks per tag set on pool... like only 2 tracks per artist
// --->Playlist selection
bRandomPick, // Get randomly from pool
probPick, // Get by scoring order but with x probability of being chosen from pool
playlistLength,
bSortRandom, // Final sorting (independently of picking method)
// --->Console
bProfile, bShowQuery, bShowFinalSelection // Different logs on console
}

Examples: Some usage examples, most of them can be combined in some way (like A with H, etc.)
A:  Random mix with only nearest tracks, most from the same decade
args = {genreWeight: 15, styleWeight: 10, moodWeight: 5, keyWeight: 10, dateWeight: 10, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, sbd_max_graph_distance: 150, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

B: Random mix a bit varied on styles (but similar genres), most from the same decade. [like A with more diversity]
args = {genreWeight: 10, styleWeight: 5, moodWeight: 5, keyWeight: 5, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, sbd_max_graph_distance: 250, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

C: Random mix even more varied on styles/genres, most from the same decade.
args = {genreWeight: 0, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 50, sbd_max_graph_distance: 300, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

D: Random mix with different genres but same mood from any date.
args = {genreWeight: 0, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 0, bpmWeight: 5, bpmRange: 25,
probPick: 100, scoreFilter: 50, sbd_max_graph_distance: 600, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

E:  Uses the properties of the current panel to set all the arguments.
do_searchby_distance();

F:  Uses a properties object to set all the arguments. [like E but overriding the properties used, usually for merged buttons]
args = {properties: yourPropertiesPairs}; // {key: [description, value], ...} see SearchByDistance_properties
do_searchby_distance(args);

G:  Outputs only influences from any date. [doesn't output similar genre/style tracks but influences! May be similar too, but not required]
args = {genreWeight: 5, styleWeight: 5, moodWeight: 15, keyWeight: 10, dateWeight: 0, bpmWeight: 10, bUseInfluencesFilter: true,
probPick: 100, scoreFilter: 40, sbd_max_graph_distance: 500, bRandomPick: true, method: 'GRAPH'};
do_searchby_distance(args);

H:  Uses the properties of the current panel, and allows only 2 tracks (n+1) per artist on the pool. [like E with pool filtering]
args = {poolFilteringTag: 'artist', poolFilteringN: 1} // bPoolFiltering evaluates to true whenever poolFilteringN != -1
do_searchby_distance(args);

I:  Random mix even more varied on styles/genres, most from the same decade. [like C but WEIGHT method]
args = {genreWeight: 10, styleWeight: 5, moodWeight: 5, keyWeight: 5, dateWeight: 25, bpmWeight: 5,  dateRange: 15,
bpmRange: 25, probPick: 100, scoreFilter: 60, bRandomPick: true, method: 'WEIGHT'};
do_searchby_distance(args);

J:  Mix with similar genre/styles using DYNGENRE method. The rest of the args are set according to the properties of the current panel.
args = {dyngenreWeight: 20, dyngenreRange: 1, method: 'DYNGENRE'};
do_searchby_distance();*/

Code: [Select]
/*	TODO:
- Replicate progressive lists and intelligent list ordering, saving moods and genre/styles not in common.
-Then selecting tracks for final playlist according to those values; like 4 tracks with set (A,B,C), then change 1 variable, then another 4 tracks, etc.
-Not grouping instrumental tracks if possible, etc.
- Clean unused things
- Fingerprint comparison for scoring?
- Output just similar tracks by fingerprint
- Give more weighting to similar tracks
- Fine-tune Pre-filtering:
- Fine-tune queries
- GRAPH : Apart from individual link caching, cache entire distance from set (A,B,C) to (X,Y,Z)
- Cache all links below a threshold? Instead of caching all possible links.
- Different date formats or just TF? YYYY-MM-DD; YYYY-MM
- Get data using MusicBrainz Ids for all tracks on library, save to json files and use them when no tags present
- Support for more high-level data (https://acousticbrainz.org/data#sample-data):
- danceability
- gender
- moods_mirex
- timbre
- tonal_atonal
- voice_instrumental
- Instruments
- Make it easy to port the code to other frameworks outside foobar:
- Include are just replaced with import
- Properties are just objects
- All graph calcs and descriptors are independent of foobar so they work 'as is'. That's the main feature and heaviest part.
- All tag 'getValues' would be done using external libraries or native player's methods
- Queries would be disabled (they are not intrinsic part of the code, just a speed optimization)
- Handle lists should be replaced with similar objects
- Console logs work as they are.
- Helpers used don't need foobar at all (except the 'getValues' for tags obviously)*/

 
SimplePortal 1.0.0 RC1 © 2008-2021