Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Research at the South Pole (Read 2258 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Research at the South Pole

Some time ago, it was mentioned here at Hydrogenaudio that WavPack is being used to store measurement data from brain probes, which warrants the name BrainWavPack. Also mentioned was ld-decode, a project where FLAC is being used to compress undecoded LaserDisc rips. In a similar vein, I recently attended a conference were use of FLAC to compress video tape signals (raw, directly from the tape head) sampled at 40MHz was presented. According to the presenter (Radoslav Markov) 175GB/h worth of signal was compressed down to about 20GB/h. Presentation can be viewed here. (FLAC is mentioned at 3:00:00 in the video)

However, yesterday I was pointed to another use of FLAC, at the South Pole no less, for storing signals of a telescope. The mention of FLAC is here. As mentioned here, FLAC has been used for the SPT-3G and is planned for the CMB-S4 project.

I find this fascinating. If anyone knows of other projects (I've seen mentions not referring particular projects) it would be great to share  :D

edit: here's an article going into more detail on SPT-3G

Quote
After each observation, detector data are compressed by a factor of ∼8 using the lossless FLAC compression algorithm 39 and merged with the housekeeping, calibration, and pointing information to form the raw data that is the input to the data analysis and mapmaking pipeline. The SPT-3G observing cadence results in about 300 GB of compressed data stored to disk daily.

I can imagine getting such amounts of data from the south pole to somewhere else is a little more challenging than usual.
Music: sounds arranged such that they construct feelings.

Re: Research at the South Pole

Reply #1 –
South Pole is covered by Starlink IIRC...
(also, weren't satellite communications always bandwidth favoured at the expense of latency...)

Re: Research at the South Pole

Reply #2 –
Yes, since last year, but it wasn't for the few years before that. SPT-3G has been operating since 2018.

Also, CMB-S4 will need to move 4PB a year from the South Pole alone, even with help of FLAC. That is a data rate of 3.6Tbit/s, which I don't think Starlink can handle. They'll probably ship a crate of tapes or harddisks every now and then.
Music: sounds arranged such that they construct feelings.

Re: Research at the South Pole

Reply #3 –
Haven't you errored in calculations? I get 1Gb/s for 4PB/yr... That would make it doable if Starlink can be aggregated...
They get ~4-8x compression with FLAC - a lot of digital silence?
I wonder how much more could be squeezed with dedicated predictors or overall dedicated tuning or whole codec.

Re: Research at the South Pole

Reply #4 –
Yes, you are right. It is 3.6Tbit/h, which is 1Gbit/s. Very wrong indeed. Still, a lot of data.

Anyway, small signals that don't change much can be compressed quite a lot. The thing is: they aren't even using LPC it seems, but in fact FLAC preset 0: with this amount of data, speed is everything. They don't care about a file that is 10% bigger, but they do care about an encoder that runs 4 times as fast.
Music: sounds arranged such that they construct feelings.

Re: Research at the South Pole

Reply #5 –
It isn't at all clear that all will be sent as collected. CMB-S4 will have "4 PB storage server at South Pole for 1 year of data", and also SPT-3G leaves a lot of data locally until the summer:

Quote
In addition to the full
downsampled data set, about half of the full-rate (compressed)
data are also transferred north via TDRS each day; the rest of
the full-rate data remain on local storage at the South Pole until
they can be shipped out during the austral summer season each
year.

Re: Research at the South Pole

Reply #6 –
I just read some news that made me think of this topic. Apparently SpaceX has released pricing info for 'uplink', so things that need capacity without overbooking 24/7. See here.

Quote
Starting at $75,000/Gbps/month with a one-time upfront cost of $1,250,000.

So, that upgraded telescope, the CMB-S4, outputs data at 1Gb/s, with compression. Without it, output would be 4 to 8 times as large. So, if they would indeed use Starlink in the future, using compression would save them at least 225.000$ each month I presume?

Not sure whether that pricing would be applicable to the south pole though. As getting data from the south pole would presumably need more inter-satellite hops than any other remote community on earth, the price (and savings) would be even higher. As people need to get in and out by helicopter anyway, flying harddrives in and out is probably still way cheaper.
Music: sounds arranged such that they construct feelings.