Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Using tar for storing music and backups (Read 4186 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Using tar for storing music and backups

Are there any thoughts about using tar instead of having the normal directory/audio files? I currently have one directory per disc. i.e

Some Artist - Year - Some Album (Disc 1)/track one.mp3 etc

I use foobar through wine on Linux and downloaded a component that allows reading of archives. If not tar, what about other archive formats? I see the following advantages and disadvantages:

*** Pros ***

Faster transferring of files (small files transfer slower than large ones)
Read only format (I think) - less likelihood of tags getting messed up
Less fragmentation due to less files on the hard drive

*** Cons ***

Lack of support by media players (?)
Slower seek times compared with unarchived files (?)


Anyway, if anyone has any advice or comments that would be great.

Edit: I'm suggesting each disc be tarred: Some Artist - Year - Some Album (Disc 1).tar
So there's no confusion.

Using tar for storing music and backups

Reply #1 –
Faster transferring of files: Your music is already compressed, it makes no sense putting it into an archive for transferring (when you mean not just tar, but also compressed, things like .tar.gz, it makes even less sense, since that introduces additional headers and time for decompression, etc).

Read only format: In case you want to protect your files from being written, by tagging capabilities of the player software, simply remove writing permissions. I gather from your post, that use use some UNIXoidal system, in that case: chmod -w *[mp3|ogg|flac|mp4|m4a]. Having those files inside a .tar archive introduces no level of writing security. The tar libraries often support transparent file handling of contained files in a .tar archive.

Less fragmentation: Has absolutely nothing to do with file itself. Depending on the FS, you either have less fragmentation (for instance: Ext2 and later) or, you have a lot of fragmentation, (FAT32 for instance). Fragmentation occurs at the block level, never at logical file level or something. Defragmentation sometimes even destroys data. Filesystems like Ext3 are fragmenting practically only when writing large files very slowly.

Lack of support by media players: Certainly, since it makes no sense.

Slower seek times compared with unarchived files: It would have no effect on that, except if the archive is compressed. The .tar archive might be understood like an additional layer of FS. Remember: "tar" is an abbreviation for tape archive. It is essentially a filesystem designed for linear access.

Conclusion: I don't want to sound arrogant or something, but it seems that your lack of technical knowledge about those systems made you come to the wrong solutions for - well - nonexistent problems, really. I believe, what you need is a better music library tool. When transferring files over network or on other drives, I suggest using rsync. I usually use rsync -alPvz I sometimes don't use the "z" switch, which filters files through zip compression. Since your music is already compressed, you could only count on file deduplication, which is a feature of the FS.

Using tar for storing music and backups

Reply #2 –
polemon: thanks for the detailed reply. I'm not suggesting compressed archives. Tar only. I began thinking about it after transferring ~300 mp3's to a memory stick. The transfer rate was about 2mb/sec. After I tarred them (each album), the rate jumped to about 6mb/sec. I'm sure you know more about the reasons for this. My understanding is that larger files transfer quicker and that is my experience. I'm going to do some tests myself. Cheers.

Using tar for storing music and backups

Reply #3 –
Well Windows Media Player is known to screw up your media files even if you think you are only playing (=reading), so it might be a point to have stuffed somewhere where WMP cannot reach it. Of course, that is solved as you use *n*x.

Using tar for storing music and backups

Reply #4 –
Quote from: simonh on
polemon: thanks for the detailed reply. I'm not suggesting compressed archives. Tar only. I began thinking about it after transferring ~300 mp3's to a memory stick. The transfer rate was about 2mb/sec. After I tarred them (each album), the rate jumped to about 6mb/sec. I'm sure you know more about the reasons for this. My understanding is that larger files transfer quicker and that is my experience. I'm going to do some tests myself. Cheers.


It depends how you transfer those files. Did you just use cp, or mv? or did you use something else? Try unison or rsync, it should increase transfer speed even more. Also, what filesystem do you use on the USB stick? Try using a FS that is optimized for flash media, maybe even Ext4 (which is a problem when using on Windows computers, mind you). Things like FireWire are optimized for large filestreams, but those are rather specialized, for things like camcorders, etc. where the data is stored on DV tapes, etc (which in turn is a linear data storage of course, with no real FS). You used an USB stick this time, but try an SD card next time or something, you should see a difference even when the SD card is connected via USB to your computer.

Using tar for storing music and backups

Reply #5 –
IME, it has to do with directory access, and how slow that is (relatively speaking) on USB flash drives.

The more individual files you write, the more entries have to be made in the directory, and that's what seems to slow down the writing process.

I once had a 1 GB Lexar Firefly drive that I did some tests on as it was much slower than my SanDisk flash drives when doing write or delete operations involving multiple files.

I don't remember the exact numbers now, but simply deleting several dozen files took something like 3 minutes on the Lexar, but only seconds on the SanDisks (they were all formatted as FAT32).

@polemon:  Given your comment on Ext4, I'm assuming that at least some of this behavior is due to inefficiencies in the FAT32 file system itself?
"Not sure what the question is, but the answer is probably no."

Using tar for storing music and backups

Reply #6 –
Another issue causing slow writes on USB media is that usually write caching is disabled.  This is because USB keys are often removed without flushing and dismounting the filesystem.  If you are careful to always properly dismount your USB keys first then you can enable write caching and speed things up.

Using tar for storing music and backups

Reply #7 –
Quote from: mixminus1 on
@polemon:  Given your comment on Ext4, I'm assuming that at least some of this behavior is due to inefficiencies in the FAT32 file system itself?


More likely than not. The problem is, that the flash memory used are very fast when seeking, but passing through large chunks of continuous data is rather slow, when a lot of seeking is involved. SSDs had problems with random writes since their beginning. This problem shows on USB sticks as well. Now, the problem with FAT32 is, that it is very prone to fragmenting, meaning a lot of random writes. Ext4 uses b-trees and not a FAT block. Meaning every block contains the address of the second block, so there's less seeking back to a TOC as it is with FAT (hence file allocation table). Furthermore, Ext4 has some optimizations towards flash media. I don't know exactly what is optimized there, but it's definitely geared towards flash media as in SSDs, etc.

Using tar for storing music and backups

Reply #8 –
Quote from: washu on
Another issue causing slow writes on USB media is that usually write caching is disabled.  This is because USB keys are often removed without flushing and dismounting the filesystem.  If you are careful to always properly dismount your USB keys first then you can enable write caching and speed things up.


Yeah, but then it takes forever when unmounting the drives, since those files are written to the memory. Also, when someone is uncareful enough to rip the stick out of the drive, without unmounting it (or when the cache is written only partly to non-volatile memory), it will more often than not, destroy the FS or at least cause some data loss.

Using tar for storing music and backups

Reply #9 –
I stand corrected:

*** 30 mp3's in two directories ***
simon@Jupiter:~/Test$ time cp audio -R /media/Titan/Incoming/

real    0m6.689s
user    0m0.007s
sys    0m0.433s

*** archives ***
simon@Jupiter:~/Test$ time cp audio -R /media/Titan/Incoming/

real    0m6.792s
user    0m0.008s
sys    0m0.385s
simon@Jupiter:~/Test$

Not significantly slower. Both filesystems ext4.

Using tar for storing music and backups

Reply #10 –
You probably need to run "sync" to time those commands correctly:

Code: [Select]
$ time { cp -R audio /media/Titan/Incoming/; sync; }
 

Using tar for storing music and backups

Reply #11 –
Quote from: skamp on
Code: [Select]
$ time { cp -R audio /media/Titan/Incoming/; sync; }

Hmm, when write caching is disabled, it should take zero time to sync. Nice way to quickly test if caching is disabled, actually.