Automating Audio File Changes

Topic: Automating Audio File Changes (Read 1963 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

Automating Audio File Changes

2016-02-15 16:07:10

Hi,

I have this directory full of music which is downloaded via cron & updates periodically:

/home/username/media/collection/artist/album/(part1,part2,part3,etc.)/(foo.mp3, foo.ogg, foo.wav, foo.flac, foo.aac)

I want to re-encode it all to .ogg with uniform values, & do the following:

Make bitrates ~128kbps
Rename the files with a new naming convention based on their original metadata/tags/comments
Remove original metadata/tags/comments
Add new vorbis comments based on the new filenames
Place altered files in /home/username/media/ogg/collection/foo.ogg

Then I need a script to alter new files which periodically appear in new source directories & do the same to them, without re-doing that which has already been done previously- on each pass.

I'm guessing there must be a way to do something like:

Create a file containing a list of previously altered files
Make a new file based on that file which contains a list of files which haven't been processed already
Then the script that alters things reads that file to know what to process

But I don't know how to construct such a system.

I'm running a headless Ubuntu Server 14.04 & have the following packages installed to choose from to handle this task:

libav-tools
oggconvert
dir2ogg
vorbis-tools
libvorbisenc2

The end result of all this is so that I can have a fallback playlist for Icecast2 to play when Airtime doesn't have any show(s) scheduled.

Please advise.

Thanks,
-w

Re: Automating Audio File Changes

Reply #1 – 2016-02-15 17:17:38

Are you asking for help with the tagging script? Or just the file management aspect?

For the file management, I think you could just write a list of recently added/modified files using "find" and then have your tagging script work through that. Just set cron to run it every 24 hours or whatever and find files modified in that timeframe.

If you don't know how to start writing the tagging script, then I guess you'll need to learn a lot about scripting. Investing in a decent book on Bash would probably be wise - I always liked the O'Reilly books for this kind of thing. You already have a clear set of tasks for your script to accomplish, so just take it one tiny step at a time, testing it out on a small subset of files copied to a test directory. The script will read lines from your file list, use id3tool or whatever to grab each tag, store it to a bash variable, and process it appropriately before feeding those strings to encoding tools. It's the string processing that will probably take the most work, because you'll need to escape special characters and such. Get that part done first - just have your script print the strings it will use without actually doing anything. Then work out the transcoding with ffmpeg or whatever from there.

Hope that's somewhat helpful. It's not clear to me how much you already know.

Re: Automating Audio File Changes

Reply #2 – 2016-02-15 18:33:48

Hi! & thanks for your reply.

Quote from: tppytel on 2016-02-15 17:17:38

just write a list of recently added/modified files using "find" and then have your tagging script work through that.

This is one part I think I might be able to do [1], thanks to Google & IRC. I don't know if what I've put together here will work, I just was able to find stuff on Google after getting a clue on IRC re: what I might do to accomplish this part. Correct me if I'm wrong here.

Quote

you'll need to learn a lot about scripting.

Only that which is pertinent, thank god. I'll certainly be seeking advice from #bash on freenode regarding the direction I take this. I'm not entirely sure where to begin on much of it.

Quote

one tiny step at a time,

Yes, as frustrating as the learning curve may be, of course it's a must to take baby steps.

Quote

read lines from your file list

Not sure what you mean here... I've got a handle on the pre-processing lists, although I may find I need more sed than I'm currently comfortable with- will have to tinker a bit & take it slow.

Quote

It's the string processing that will probably take the most work, because you'll need to escape special characters and such.

I imagine there are plenty of examples of work similar enough that I can pick apart stuff I find on Google & piece together something that works by means of trial & error. I just need help to have someone point me in the right direction regarding where to begin to look for such stuff is all.

Quote

just have your script print the strings it will use

I don't know what you mean by this. Are you talking about writing a script which only does a simulation & would echo the commands it would execute? I'd have no idea how to do that. Maybe you're referring to the list creation, in which case while I am worried about the relevant details regarding syntax & i/o, I'm fairly confident I can assemble something- providing I get healthy clues from those who care to reply to my posts, etc.

Of course I welcome any & all advice & direction as to how I might go about this. Thanks to everyone in advance for helping. :-)

[1]

Code: [Select]

#!/bin/sh

MONITORDIR="/home/username/media/collection/"

inotifywait --daemon --recursive --syslog --quiet --timeout 10800 --event create --format '%w%f' -o /home/username/media/log/tmp/just.pre.processed "${MONITORDIR}" | while read JustPreProcessed; do cat -- "$JustPreProcessed" >> /home/username/media/log/PreProcessed.all; done

Re: Automating Audio File Changes

Reply #3 – 2016-02-15 19:14:12

Quote from: wetyourself on 2016-02-15 18:33:48

I don't know if what I've put together here will work...

I haven't used inotify, so I'm not sure. But try it out... if you end up with a list of filenames in a text file, then you're on the right path. It's actually been quite a while since I've written a script of the complexity that you'll need, so don't take any of my advice at face value. But I've written some reasonably sophisticated scripts before, so hopefully I won't lead you too far astray.

Quote

read lines from your file list...
Not sure what you mean here...

Once you've got a list of files that need processing, you'll want your script to iterate over each file in the list. So you pull in one line of the file to $FILENAME, do your things on that file, then read the next line into $FILENAME, rinse and repeat. Everything happens inside a big "while" loop. Google "bash read file line by line" for some examples.

But you don't need to do any of this until the end. For starters, work on the rest of the tag processing and encoding and just give a single filename directly to the script on the command line. Then FILENAME=$1 assigns that argument to $FILENAME. Once you have the processing working correctly for a single file, then go back and figure out how to read lines.

Quote

I may find I need more sed than I'm currently comfortable with

Yes, sed and regular expressions are going to be important.

Quote

just have your script print the strings it will use...
I don't know what you mean by this. Are you talking about writing a script which only does a simulation & would echo the commands it would execute? I'd have no idea how to do that.

Yes, and you basically just said how to do it. Get a filename as described above. Then save the info you need into however many variables are required. Probably start with something like TAGS=`id3tool $FILENAME`, then pull the right bits out of that using grep. You'll have to figure out the exact syntax you need for that part, because I'm rusty, but basically ALBUM=`echo $TAGS | grep blahblahblah`. Then do whatever further processing tricks you need to with sed, operating on $ALBUM, $ARTIST, etc. Then have the script just echo what it thinks its information is with lines like echo "Album: $ALBUM" and so on. Once you're satisfied that you've munged all the strings the way you want them, then you figure out the encoding/transcoding and renaming parts, using those $ALBUM, $ARTIST, etc. variables in the commands.

Re: Automating Audio File Changes

Reply #4 – 2016-02-15 21:04:23

Quote from: tppytel on 2016-02-15 19:14:12

then pull the right bits out of that using grep. You'll have to figure out the exact syntax you need for that part, because I'm rusty, but basically ALBUM=`echo $TAGS | grep blahblahblah`.

Sorry, I'm not understanding how grep is of use here. As I understand it it can pull lines out which match string/regex, but I don't understand how I would use it to 'insert data' or 'change values' of said data. Could you elaborate for me maybe with an example. It could be so simple I'm missing it, after all I've been up at building this server for ~30 hours by now, hehe. Thanks...

Re: Automating Audio File Changes

Reply #5 – 2016-02-15 23:04:35

Quote from: wetyourself on 2016-02-15 21:04:23

...but I don't understand how I would use it to 'insert data' or 'change values' of said data. Could you elaborate for me maybe with an example.

The snippet you quoted is most of the idea. You just have to work out the implementation details. It sounds like you're not understanding the concept of shell variables and/or not understanding backticks. When bash sees a line like

Code: [Select]

ALBUM=foo

it assigns the value "foo" to the variable "ALBUM". Then you can use that value in further commands by prefixing $. So

Code: [Select]

echo $ALBUM

just returns "foo".

Backticks indicate command substitution. Bash does whatever you tell it inside the backticks and uses the return value. So if you write

Code: [Select]

ALBUM=`echo "Hello World"`

then ALBUM now has the value "Hello World". You can test all of this on the command line too while you're learning... no need to write an actual script.

You have a variety of command-line tools for different file types (FLAC, MP3, Ogg, etc.) which all read tag data and spit it back out to you in different ways. For example, metaflac looks like...

Code: [Select]

tppytel@calliope:~/test$ metaflac --list --block-type=VORBIS_COMMENT example.flac
METADATA block #2
  type: 4 (VORBIS_COMMENT)
  is last: false
  length: 505
  vendor string: reference libFLAC 1.2.1 20070917
  comments: 17
    comment[0]: TITLE=We See
    comment[1]: ACCURATERIPID=0018ddbc-00d9d0e4-9f0fa00b
    comment[2]: ACCURATERIPCRC=d48847a4
    comment[3]: ACCURATERIPDISCID=011-0018ddbc-00d9d0e4-9f0fa00b-01
    comment[4]: ACCURATERIPCOUNT=30
    comment[5]: ACCURATERIPCOUNTALLOFFSETS=30
    comment[6]: ACCURATERIPTOTAL=42
    comment[7]: TOTALTRACKS=7
    comment[8]: Album=Monk
    comment[9]: Artist=Monk, Thelonious
    comment[10]: Comment=2000 remaster from The Complete Prestige Recordings
    comment[11]: Genre=Jazz
    comment[12]: ALBUMARTIST=Monk, Thelonious
    comment[13]: LABELNO=3PRCD-4428-2
    comment[14]: DATE=2000
    comment[15]: TRACKNUMBER=01
    comment[16]: ORIGINAL YEAR=1954

You need to get the right parts of that output assigned to the right shell variables using some combination of pipes, temp files, grep, sed, whatever. That's where the work is, and it's going to be a little different for each file type, so you'll need to check the type first and use if-then's to branch appropriately. In this case I'd probably dump the tag data to a temp file and then process it from there...

Code: [Select]

metaflac --list --block-type=VORBIS_COMMENT example.flac > tags.txt
ALBUMARTIST=`cat tags.txt | sed -n -e 's/^.*ALBUMARTIST=//p'`

Now ALBUMARTIST is "Monk, Thelonious" and I can use $ALBUMARTIST later on in the script when writing new tags, encoding, creating filenames, etc.

Again, you're best off learning this stuff by just playing around directly on the command line. Copy a single file into a temporary directory so you don't accidentally mess anything else up and experiment with how you can read tag values with the command-line tools and get them assigned to shell variables. When you're assigning values to variables on the command line, it's probably good practice to do

Code: [Select]

export FOO=bar

instead of just

Code: [Select]

FOO=bar

The export command adds the variable to your current environment and makes that value available to subprocesses, which is typically what you want when you're testing. You can then see the list of your current environment variables by running "env". You can remove existing assignments with "unset FOO", etc.

Re: Automating Audio File Changes

Reply #6 – 2016-02-15 23:13:23

Quote from: tppytel on 2016-02-15 23:04:35

Quote from: wetyourself on 2016-02-15 21:04:23
...but I don't understand how I would use it to 'insert data' or 'change values' of said data. Could you elaborate for me maybe with an example.
The snippet you quoted is most of the idea. You just have to work out the implementation details. It sounds like you're not understanding the concept of shell variables and/or not understanding backticks. When bash sees a line like
Code: [Select]
ALBUM=foo
it assigns the value "foo" to the variable "ALBUM". Then you can use that value in further commands by prefixing $. So
Code: [Select]
echo $ALBUM
just returns "foo".

Backticks indicate command substitution. Bash does whatever you tell it inside the backticks and uses the return value. So if you write
Code: [Select]
ALBUM=`echo "Hello World"`
then ALBUM now has the value "Hello World". You can test all of this on the command line too while you're learning... no need to write an actual script.

You have a variety of command-line tools for different file types (FLAC, MP3, Ogg, etc.) which all read tag data and spit it back out to you in different ways. For example, metaflac looks like...
Code: [Select]
tppytel@calliope:~/test$ metaflac --list --block-type=VORBIS_COMMENT example.flac
METADATA block #2
  type: 4 (VORBIS_COMMENT)
  is last: false
  length: 505
  vendor string: reference libFLAC 1.2.1 20070917
  comments: 17
    comment[0]: TITLE=We See
    comment[1]: ACCURATERIPID=0018ddbc-00d9d0e4-9f0fa00b
    comment[2]: ACCURATERIPCRC=d48847a4
    comment[3]: ACCURATERIPDISCID=011-0018ddbc-00d9d0e4-9f0fa00b-01
    comment[4]: ACCURATERIPCOUNT=30
    comment[5]: ACCURATERIPCOUNTALLOFFSETS=30
    comment[6]: ACCURATERIPTOTAL=42
    comment[7]: TOTALTRACKS=7
    comment[8]: Album=Monk
    comment[9]: Artist=Monk, Thelonious
    comment[10]: Comment=2000 remaster from The Complete Prestige Recordings
    comment[11]: Genre=Jazz
    comment[12]: ALBUMARTIST=Monk, Thelonious
    comment[13]: LABELNO=3PRCD-4428-2
    comment[14]: DATE=2000
    comment[15]: TRACKNUMBER=01
    comment[16]: ORIGINAL YEAR=1954
You need to get the right parts of that output assigned to the right shell variables using some combination of pipes, temp files, grep, sed, whatever. That's where the work is, and it's going to be a little different for each file type, so you'll need to check the type first and use if-then's to branch appropriately. In this case I'd probably dump the tag data to a temp file and then process it from there...
Code: [Select]
metaflac --list --block-type=VORBIS_COMMENT example.flac > tags.txt
ALBUMARTIST=`cat tags.txt | sed -n -e 's/^.*ALBUMARTIST=//p'`
Now ALBUMARTIST is "Monk, Thelonious" and I can use $ALBUMARTIST later on in the script when writing new tags, encoding, creating filenames, etc.

Again, you're best off learning this stuff by just playing around directly on the command line. Copy a single file into a temporary directory so you don't accidentally mess anything else up and experiment with how you can read tag values with the command-line tools and get them assigned to shell variables. When you're assigning values to variables on the command line, it's probably good practice to do
Code: [Select]
export FOO=bar
instead of just
Code: [Select]
FOO=bar
The export command adds the variable to your current environment and makes that value available to subprocesses, which is typically what you want when you're testing. You can then see the list of your current environment variables by running "env". You can remove existing assignments with "unset FOO", etc.

Hey, thanks! with help like yours, I'm well on my way...

Re: Automating Audio File Changes

Reply #7 – 2016-02-16 03:23:16

Good. You'll end up learning scripting just like everyone does - you have a task that needs doing, and you research more and more complicated tricks as you need them. You're jumping in with a fairly large scripting task - especially given that you need to work with multiple tag formats - but it sounds like you'll get there.

Some other things you'll want to watch out for...

Special characters in tag values are going to be a PITA if you're going to convert those into filenames. Certain characters like / and ? absolutely must be removed from filenames. You probably want to remove anything remotely weird just to be on the safe side - when you start feeding all those names through different programs, it's hard to predict what might cause a problem. Sed will be your friend there... use lines that assign a variable back to itself after processing like ALBUM="`echo $ALBUM | sed blahblahblah`".

Multi-line or multi-field comments could be annoying, if you use those. My main library, for example, uses multiple values in the ARTIST and ALBUMARTIST fields - Media Monkey understands this, but the script will need to make some decisions about what value to use.

You'll want to do some sanity-checking of those strings. Probably you have at least a handful tracks floating around with missing album artist tags or whatever. Make sure your script does something predictable if a tag field is blank.

You may also need to deal with variations in the tag format. As you can see in the output I pasted above, the artist tag was single-cap "Artist" but the album artist was all-caps "ALBUMARTIST" because MM writes tags in a different way than MP3Tag. If your library has been built up over many years from a variety of sources, it's going to have relics of earlier software/procedures in there.

Notice