Skip to main content
Topic: Converting a HUGE collection (Read 16070 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Converting a HUGE collection

Like many of you, I was ripping and encoding Mp3's many years ago.
And although I never used Xing, I can't say my 128 bit Blade rips are making me happy today.

I have come to a crossroads, and am not sure which path to take. On one hand, I am really happy with how LAME -aps sounds, and of course like the size. (Of course, I remember saying the same thing 5 years ago. Will I say it again in 5?)

On the other hand, I can go the lossless route and be assured of a great archive.

My problem is that I have just over 10,000 cd's to convert. The storage requirements in going with something like FLAC is quite large. I estimate only getting about 380 albums on a single 120GB drive (around $90).  DVDR media actually comes close to the same cost. And CDR format was already out of hand for MP3 size (1200 CDR's stacked up).

To me, I feel like the only option is to wait until 1TB drives cost $99. (Not going to happen) or got with LAME -Aps.

Converting a HUGE collection

Reply #1
10000 cd's .. wow..

aps seems to stack up pretty well, no complaints from me, wonder how much space it will take for that kind of data @ ~ 200 kbps .. prolly more that 120 gig, much more

Converting a HUGE collection

Reply #2
MPC is a great middle ground for you that has quality arguably better than mp3, but with comparable sizes.  You might want to give that a try.

Just as a reference, you have over 10,000 CDs.  What are you, a DJ?  That's over $100,000 spent on CDs.  If you drop another $1000 or so, you could easily get a 1TB Raid-5 array for yourself. 

Also, as another reference, assuming they're all full-length music CDs, you have about a year's worth of music sitting in CD form.


Sorry, didn't mean to be offensive, but that's a LOT of CDs.  I have a hard time visualizing 10,000 of something like that.

Converting a HUGE collection

Reply #3
By the time you'll be halfway through ripping, chances are the available HD space is tripled and the cost is a third of what it is today! 

My solution is letting Musepack shrink the vast bulk of the CD's that pass me by, and keep the most important ones (for any reason) losslessly compressed with FLAC. Every time I reevaluate my decision (which is sometimes too often) I end up being quite happy about it.

Cheers,
Uosdwis

Converting a HUGE collection

Reply #4
I only have about 2500 CDs and I'm already in such a conundrum, but I'm not as bad off as you, of course  I just need to pick up a 250GB to hold all my LAME rips (aps) plus plenty of movie rips.

50minutes/album avg. * 60 seconds/min * 200kilobit/sec  /8 byte/bit  = ~75MByte/album
2500 albums at around 75MB per is roughly 187Gigs, and I have 80 non-OS, which is stuffed with the rips I haven't burned yet

You're looking at closer to three-quarters of a Terabyte with aps rips 

If archiving is your primary goal with ripping, and you're largely going to be playing the rips from your computer, it seems like MusePack/MPC would be the best option for quality and size.

  The ideal option is probably MPC on a large RAID 1 array just in case one of the Hard drives takes a dive. Which happened to my first 30Gigs of rips 
  Edit because I dont rip movies with alt-presets  and because RAID 0 is not redundant

Converting a HUGE collection

Reply #5
To think I was getting rather proud of my 300 cd collection... sigh...

at least I can say they are all really good discs!

Converting a HUGE collection

Reply #6
yeah, I got interested in FLAC. Did a quick test. Some of my old Rush CD's came out around 250MB. OK, not so bad. That might work.

Vapor Trails came out around 500MB.  I figure the middle ground would be around 350MB. That just won't work when you're talking about archiving 1000 (or more) CD's.
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

Converting a HUGE collection

Reply #7
Thanks for the nice comments, everyone. That's why this site is the best!

sthayashi, as Audible mentioned, a TB worth of storage will be needed just to house that much as simply MP3-aps. That is certainly affordable for me (although pushing it), although I'd probably wait until the next size up from 120GB breaks the $99 mark, and with Serial-ATA. But contemplating the cost of hard drives to hold the Lossless -- no way.

I think the only reason I was even thinking about lossless archiving as a solution was because when I did the math, it looked like 8 or 9 FLAC albums would fit on a DVDR.  This of course is close to the ratio of mp3 albums fitting on a CDR.  Which back in the day, I didn't have any complaints about! (I don't know if I'm alone in this, but discovering mp3's back in 1998 was kind of like having a childhood Christmas, a Birthday, and sex for the first time, all rolled into one!)

The 1200 CDR's I have stacked up in spindles doesn't bother me even now, so would 1200 DVDR's bother me? Not if the price was the same as CDR's are. That'll take a while, but can I get 4X DVDR for $1 each now? That seems affordable at staggered intervals (which luckily, this hobby takes a while to do)

I've recently started my education on MPC, so will have to ponder that.
My first mistake was looking at the main web site of it (all black background, millions of useless links)
Since computer playback is the main use here, MPC sounds like a sound fit. However, the popularity of MPC may hamper my style. Not that...*cough*....I'd ever trade these... *cough*... But if some other Slider's World I did, MPC might not be any good.

 

Converting a HUGE collection

Reply #8
Concerning the cost of hard disk capacity, I thought I'd add something...

I bought my first *real* PC in college (not counting a Commodore VIC20 as a *real* PC, but it was fun!)  It was 1988, and the machine was one I bought the pieces for and assembled myself, component-by-component.

Here are the specs:

Intel 286 / 16 MHz
1 MB RAM --- 640 kB was pretty standard at the time, but I sprung for the *big* memory option 
40 MB HDD
5 1/4" floppy --- giant disks which stored 320 kB to 640 kB 

...well, you get the picture.

Anyway, that 40 MB hard disk cost me about $300!!!  That comes out to $7.50 per MB.

-----------------------------------------------------------------------------------------------

Fast-forward to 2003.  Just for a comparison of new technology, my portable MP3 player takes Memory Sticks (flash memory) which are smaller than a stick of gum and hold 32 MB up to 1 GB.  The biggest I own are 128MB, which now run about $50 each (~$0.40 per MB).

As you stated, a 120 GB hard drive now costs about $90.  That's $0.00075 (75/1000 of one cent) per MB.

So we've seen a price drop of some 1,000,000%.  A ONE MILLION PERCENT price drop per MB of HD storage in 15 years...or an average of 66,667% per year.

Now, price fluctuations aren't consistant year-to-year, so if you made a realistic graph, the the curve would be pretty bumpy.  Hence, you couldn't apply the same formula to a small time frame (<5 years)...so instead...

-----------------------------------------------------------------------------------------------

Fast forward to 2018...fifteen years in the future.

If the same formula applies over then next fifteen years, then HD storage space will cost 1/1,000,000% of what it does today.  That comes out to $0.000000075 per MB, or $0.000075 per GB...or $0.075 (seven-and-one-half cents) per TeraByte...or $75 per 1000TB.  (OK, so what do you call 1000TB???  A MegaGoogleByte?  Actually I think it's a "QuadraByte", but I'm just guessing.)

-----------------------------------------------------------------------------------------------

Anyway, my point is that you can count on the cost of HD storage dropping DRAMATICALLY in the coming years, with the biggest price drops hitting an existing technology right when a new technology hits the market.

So whether that *next* technology will be holographic memory, or quantum laser static buffers, or Star Trek antimatter matrices...the effect will be the same, those spinning disks we call "hard drives" will become as cheap as dirt.

Just something to think about when you consider audio compression technologies going forward.

-----------------------------------------------------------------------------------------------

And BTW...that's a H*U*G*E collection you have...puts my ~350 CDs to shame 

Converting a HUGE collection

Reply #9
10000 cds?
AVG ~45 minutes and 800bits/sec would mean 2.7TB which would need 12 250GB drives

Stay away from 120GB drives, might save a little with $/GB for the drives but you'll need more space, raid controllers, and just wiring is gonna be a mess .

250GB SATA drives, on 2 raid cards able to handle 8 drives.

You could just get 16 drives at once and leave alot of room for expansion, it'd be a fixed cost  just gotta go all out at once hehe.

I'd wait a litlte longer till 250GB sata drives drop below $200 (you can find pata ones under 200 but dunno bout sata)  and also check out sata raid cards, havn't looked into it much.  SATA supports hotswap and all and if a drive ever failed it'd be nice to have a card that supports it and would make it easy to rebuild the array.


Using MPC would save you alot of space / money but in it also costs you your time, if you ever rerip them all again to lossless in the future, cost of your time would pay to do it now


---edit---

"5 1/4" floppy --- giant disks which stored 320 kB to 640 kB "
hehe I dunno if its funny or sad that a 5.25 floppy needs an to be explained

Converting a HUGE collection

Reply #10
Its gonna be quicker encoding 10,000 cds as MPC instead of lame aps. 10,000 is ALOT of cds, you gonna rip these one by one? that would take forever !! 

Converting a HUGE collection

Reply #11
Quote
My problem is that I have just over 10,000 cd's to convert.

10.000!?!?!?! didn't you mean 1000 ... 10.000 CDs is about 200.000 dollars on CDs only!

Converting a HUGE collection

Reply #12
I've enjoyed reading this thread since I'm going through something similar but not on the same scale.

I just bought 4x120GB UDMA drives on sale for ~$80USD ea and built a cheap server using RAID 0+1 & a TB Santa Cruz (also on sale for $40) specifically so I could re-rip ~600 CDs. I'm about 200 CDs into it.

I mulled over this question for some time, lossless vs MPC or Ogg and finally decided on MP3 lame 3.90.3 -aps. While the others may be technically superior MP3 is good enough for my ears and I can play MP3s on HW devices such as my RioVolts, DVD player, and other players I'll prolly buy in the future.

What I underestimated was the time it takes to rip lotsa CDs if you want to be as accurate as possible w/ the tagging info. I'm a bit anal about that.

I haven't done it yet, but I'll prolly MP3Gain the whole collection when done. I'm hoping by the time I'm done SnelG will have included support for the more widely HW supported ID3V2 tag in MP3Gain vs the technically superior APE tag - but that's another topic  (No, I don't do silly things w/ ID3V2 like include album art - only the fields EAC fills in for me.)

Happy ripping...
Walkman

Converting a HUGE collection

Reply #13
Quote
Using MPC would save you alot of space / money but in it also costs you your time, if you ever rerip them all again to lossless in the future, cost of your time would pay to do it now
Very good point.  FLAC is also fast, under 10 minutes per CD to rip and encode on my Celeron 1.3 music server using EAC + FLAC.

Quote
Vapor Trails came out around 500MB. I figure the middle ground would be around 350MB. That just won't work when you're talking about archiving 1000 (or more) CD's.
  Nope, at $1/GB you can FLAC 1,000 CDs for about $350 USD plus either a spindle of DVD-Rs or another $350 for backup drives.

I went the FLAC route (750 CDs so far, another 400 to go) so that I'd never have to rip again, and so that I could transcode to other formats as needed without taking a second drop in quality from lossy-to-lossy transcoding.

Converting a HUGE collection

Reply #14
Quote
I went the FLAC route (750 CDs so far, another 400 to go) so that I'd never have to rip again,


That's very nice and all, but are you making dupes of your burns, using (add'l) error correction, or just assuming that the burns will last forever?

My experiences with data archiving on CD-Rs have been spotty at best, even with only Taiyo Uden or Ricoh Platinum media, let alone some of the crap I've bought at electronics and computer stores.

  In fact, I'd go so far as to say that a dedicated RAID 1 array will probably result in significantly less data loss over 5 years. And, you don't have to spend the time burning a thousand or more CD's only to see some fail....

Converting a HUGE collection

Reply #15
Again, I'm one thinking of this on a smaller scale.  I've had enough CD-R's corrupt themselves on me to not even think of using DVD-R.

I think raid1 or 5 is the solution, as one drive failing doesn't mean any data is lost, so it saves time and effort

my 150cds = 90gb = not a problem

10000cds = 6TB!  (or 3.5TB as FLAC)

And the music industry sais people aren't buying enough cd's
< w o g o n e . c o m / l o l >

Converting a HUGE collection

Reply #16
Unfortunately for parity-based RAID schemes (like RAID 5), I believe you typically want to invest in an uninterruptible power supply (UPS) if you value your data. Other than that, RAID 5 would be absolutely ideal, assuming you can afford the drives and buy a decent controller.

  Mirroring (RAID 1) doesn't have this problem of course.

Converting a HUGE collection

Reply #17
Why would Raid5 need UPS?  I thought you could recover your data with 1 drive dead as with raid1, never saw anything about power issues? :S
< w o g o n e . c o m / l o l >

Converting a HUGE collection

Reply #18
Go with western digitals special edition drives with a 3 year warrenty. SCSI drives have 5 year warrentys.

Converting a HUGE collection

Reply #19
Here are some links that may be useful. I got them off Slickdeals.net. They are Western Digital 200GB for $115 or if you prefer Maxtor 200GB for $130. These prices are after large rebates, but they are the best prices for high capacity drives that I have ever seen. Both these drives are high performance 7200 rpm drives with 8mb cache. I know Western Digital provides 3 year warranties for their 8mb drives. Not sure about Maxtor though. The Maxtor drives are quieter than the Western Digital drives once you enable acoustic management.
Anish


Edit: To get the $115 on the Western Digital, see the SlickDeals website. With minimum effort, it is $270 - $120 mail-in rebate. The Maxtor is $300 - $50 instant rebate - $120 mail-in rebate.

Converting a HUGE collection

Reply #20
Quote
10000 cds?
AVG ~45 minutes and 800bits/sec would mean 2.7TB which would need 12 250GB drives

Stay away from 120GB drives, might save a little with $/GB for the drives but you'll need more space, raid controllers, and just wiring is gonna be a mess .

250GB SATA drives, on 2 raid cards able to handle 8 drives.

You could just get 16 drives at once and leave alot of room for expansion, it'd be a fixed cost   just gotta go all out at once hehe.

I'd wait a litlte longer till 250GB sata drives drop below $200 (you can find pata ones under 200 but dunno bout sata)  and also check out sata raid cards, havn't looked into it much.  SATA supports hotswap and all and if a drive ever failed it'd be nice to have a card that supports it and would make it easy to rebuild the array.


Using MPC would save you alot of space / money but in it also costs you your time, if you ever rerip them all again to lossless in the future, cost of your time would pay to do it now


---edit---

"5 1/4" floppy --- giant disks which stored 320 kB to 640 kB "
hehe I dunno if its funny or sad that a 5.25 floppy needs an to be explained


Edit:  OK...I need help using the "Quote" feature...just ignore the quote above....that's not what I meant to grab.   

HERE'S what I meant to start with...
Quote
"5 1/4" floppy --- giant disks which stored 320 kB to 640 kB "
hehe I dunno if its funny or sad that a 5.25 floppy needs an to be explained


It's just amazing how fast the technology's moving, that's all.  I've met people recently who didn't know what a 5 1/4" floopy was.  In fact...3 1/2's are almost going away...when was the last time you put a floppy disk of any kind in your PC?

Quote
I mulled over this question for some time, lossless vs MPC or Ogg and finally decided on MP3 lame 3.90.3 -aps. While the others may be technically superior MP3 is good enough for my ears and I can play MP3s on HW devices such as my RioVolts, DVD player, and other players I'll prolly buy in the future.


I'm heading in that direction, too, I think.  When I first came to HA several days ago, I was of the state of mind that "FLAC or 320kbps MP3 are the only (compatible) compressed formats for me.  *I* can hear the difference with the lower stuff!"  But now, just a few days later, I' thinking that anything above --aps is placebo to my ears.  Gotta hand it to Dibrom for tweaking an MP3 encoder to the point where it's most basic setting is transparent for so many people.

And compatibility is what's keeping me away from other formats for now.  Until my portable device and the DMS in my car can do MPC or any lossy codec other than MP3, then all I can consider are the formats that I have players for (other than my PC).

Converting a HUGE collection

Reply #21
Quote
If the same formula applies over then next fifteen years, then HD storage space will cost 1/1,000,000% of what it does today.  That comes out to $0.000000075 per MB, or $0.000075 per GB...or $0.075 (seven-and-one-half cents) per TeraByte...or $75 per 1000TB.  (OK, so what do you call 1000TB???  A MegaGoogleByte?  Actually I think it's a "QuadraByte", but I'm just guessing.)

To keep you going over the next few years:
  • 1,000TB = 1PB = Petabyte.


  • 1,000PB = 1EB = Exabyte.


  • 1,000EB = 1ZB = Zettabyte.


  • 1,000ZB = 1YB = Yottabyte.

Although of course when most people talk in these units they really mean TiB, PiB etc, meaning it's 1024.
Quote
Anyway, my point is that you can count on the cost of HD storage dropping DRAMATICALLY in the coming years, with the biggest price drops hitting an existing technology right when a new technology hits the market.

Well, maybe.  Depends on how much further areal densities can be pushed reliably and cheaply, and how long suitable replacements will take to develop.  We're seeing a nice drop at the moment because manufacturers are moving to 80GB platters, but we're also seeing access times going up as heads need to seek more precisely, and yields dropping as smaller and smaller defects become problematic.  Better keep our fingers crossed
Quote
And BTW...that's a H*U*G*E collection you have...puts my ~350 CDs to shame  

And my, er, ~5 CDs to shame.  *sob*

Converting a HUGE collection

Reply #22
Quote from: DaveSimmons,Jul 4 2003, 06:30 PM
  Nope, at $1/GB you can FLAC 1,000 CDs for about $350 USD plus either a spindle of DVD-Rs or another $350 for backup drives.

[/quote]
Exactly. That doesn't work for me.
flac>fb2k>kernel streaming>audiophile 2496>magni>dt990 pro

Converting a HUGE collection

Reply #23
Quote
To think I was getting rather proud of my 300 cd collection... sigh...

at least I can say they are all really good discs!

No kidding, eh? As a 19-year old, I think I've done okay in having ~100 CDs. Then, to hear someone come on here and have my collection size *SQUARED*(!!) just makes me feel small.

Converting a HUGE collection

Reply #24
Quote
> I went the FLAC route (750 CDs so far, another 400 to go) so that I'd never have to rip again,

That's very nice and all, but are you making dupes of your burns, using (add'l) error correction, or just assuming that the burns will last forever?
  No, my FLACs are on 2 Maxtor 250 GB drives, and I have a couple of WD drives to use for backing them up (with the WDs kept separately from the server).

I figure I can keep copying drives with a very low chance of 2 drives (primary + backup) failing at the same time.  Current cost for this scheme ("manual RAID 1"?) is about $0.60 USD per disc, but within 2 years it will probably drop to 0.30, then 0.15, etc.

Comparing to real RAID-1:
Pro - Simpler to setup and maintain, use motherboard IDE or cheap non-RAID card
Pro - backup drives are not in PC, safer from theft, power problems, infection, or other catastrophic server failures.
Con - must remember to do manual backups.  I use my 120 GB OS drive as a temporary backup before copying to the real backup drive.
Con - must periodically test the backup drives to make sure none have failed.

$0.60 per CD is reasonable to me to have all of my CDs online (a super CD changer), in true CD quality, and both transcodable and "clonable" as needed.

 
SimplePortal 1.0.0 RC1 © 2008-2019