Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Why m3u8 loading is so slow? (Read 419 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Why m3u8 loading is so slow?

One of the problems of the Playlist Manager is that FPL playlist can not be saved to a path without using the UI (waiting for a SMP update) and m3u8, m3u playlist are really really slow as soon as you add a few hundreds of files. Takes easily +50 secs.

I get the the argument "that's why FPL exists, you have the tags there on binary, etc. blablaba"
But that's just half of the truth. My reasoning was... well if you add arbitrary files to an m3u8 playlist and then you load it on foobar, right... foobar has to load the files and read their tags. But what about files already present on the library?

It's ironic because one of the examples usually given to prove the FPL speed optimization is: "Fill a playlist with 1000 items from your library, save as fpl and as m3u8. Then close the playlist and load both files. Which one is faster?" The problem is clear, if those files are in the library... why the hell foobar doesn't compare the paths of the m3u8 files against the paths of the library items to speed up the process?

So I tried doing that in SMP, and suddenly, when all tracks are matched, loading speed goes from a few minutes for 2700 tracks (at least it's asynchronous) to 1.3 secs. SMP has a limit though, I can not mix found tracks with not found ones, since there is no method to load tracks as handles, only plman.AddLocations() which sends things directly to a playlist (another limitation reported).

Code: [Select]
const writablePlaylistFormats = new Set([".m3u",".m3u8",".pls"]); // These are playlist formats which are writable

function _isFile(file) {
if (isCompatible('1.4.0')) {return utils.IsFile(file);}
else {return isString(file) ? fso.FileExists(file) : false;} //TODO: Deprecated
}

function getFilePathsFromPlaylist(playlistPath) {
let paths = [];
const extension = isCompatible('1.4.0') ? utils.SplitFilePath(playlistPath)[2] : utils.FileTest(playlistPath, "split")[2]; //TODO: Deprecated
if (!writablePlaylistFormats.has(extension)){
console.log('getFilePathsFromPlaylist(): Wrong extension set "' + extension + '", only allowed ' + Array.from(writablePlaylistFormats).join(', '))
return paths;
}
if (_isFile(playlistPath)) {
// Read original file
let originalText = utils.ReadTextFile(playlistPath).split('\r\n');
if (originalText !== undefined && originalText.length) {
let lines = originalText.length
if (extension == '.m3u8' || extension == '.m3u') {;
for (let j = 0; j < lines; j++) {
if (!originalText[j].startsWith('#')) {paths.push(originalText[j]);}
}
} else if (extension == '.pls') {
for (let j = 0; j < lines; j++) {
if (originalText[j].startsWith('File')) {
// Path may contain '=' so get anything past first '='
let path = originalText[j].split('=').slice(1).join('='); // fileX=PATH
paths.push(path);
}
}
}
}
}
return paths;
}

// Loading m3u, m3u8 & pls playlist files is really slow when there are many files
// Better to find matches on the library (by path) and use those! A query approach is easily 20x times slower
function loadTracksFromPlaylist(playlistPath, playlistIndex) {
let test = new FbProfiler('Group #1');
let bDone = false;
const filePaths = getFilePathsFromPlaylist(playlistPath).map((path) => {return path.toLowerCase()});
const playlistLength = filePaths.length;
let handlePlaylist = [...Array(playlistLength)];
let poolItems = fb.GetLibraryItems().Convert();
let pathPool = new Map();
let filePool = new Set(filePaths);
poolItems.forEach( (handle, index) => {
if (filePool.has(handle.Path.toLowerCase())) {
pathPool.set(handle.Path.toLowerCase(), index);
}
});
let count = 0;
for (let i = 0; i < playlistLength; i++) {
if (pathPool.has(filePaths[i])) {
handlePlaylist[i] = poolItems[pathPool.get(filePaths[i])];
count++;
}
}
if (count == filePaths.length) {
handlePlaylist = new FbMetadbHandleList(handlePlaylist);
plman.InsertPlaylistItems(playlistIndex, 0, handlePlaylist);
bDone = true;
}
test.Print();
return bDone;
}

Then do:

Code: [Select]
loadTracksFromPlaylist(yourPlaylist.m3u8, plman.ActivePlaylist)

Now tell me m3u8 is always slow because blablabla. This clearly points to a bad implementation of playlist loading, since comparing files by path first clearly reduces loading time for plain text playlists. (And lets be serious, people usually has playlist for things in their library).

X

If that's in js, I can imagine it will be even better on C++ directly on the core.

EDIT: for comparison do this and take tea while the program loads +2000 tracks... (on the other hand, it only takes a ridiculous time the first time you load the playlist. Consecutive loads of the same file are fast)
Code: [Select]
plman.AddLocations(plman.ActivePlaylist, yourPlaylist.m3u8, true);

Re: Why m3u8 loading is so slow?

Reply #1
Is there any speed difference if you remove SMP from the equation and just load the m3u directly with File -> Load Playlist?

Re: Why m3u8 loading is so slow?

Reply #2
Is there any speed difference if you remove SMP from the equation and just load the m3u directly with File -> Load Playlist?
No. That's exactly equivalent to plman.AddLocations(). Tested both anyway.
And that's the point, the current implementation to load files is far from optimal (apart from the SMP missing methods reported). And that was the "main" reason to use a binary format for playlists.

Re: Why m3u8 loading is so slow?

Reply #3
M3U8 doesn't support caching subsong references, and while the Media Library does support caching that full info, if it's not in the Media Library, it will need to re-parse the file to determine the subsong count.

Re: Why m3u8 loading is so slow?

Reply #4
M3U8 doesn't support caching subsong references, and while the Media Library does support caching that full info, if it's not in the Media Library, it will need to re-parse the file to determine the subsong count.
That's true and it's one of the improvements of fpl playlists. But to be honest, that happens because developers are against M3U8 implementations of extra tags for foobar, since sub-track handling and other similar things are implemented in other fields and programs perfectly using m3u8 (like tv).

Anyway that has nothing to do with the loading process being totally sub-optimal, which is the main point of the thread (not the m3u8 limitations)

 
SimplePortal 1.0.0 RC1 © 2008-2021