Data Storage and Backups
Light microscopes, especially those used for time-lapse microscopy or super-resolution imaging, can generate huge amounts of data. In order that the core microscopes continue to function it is necessary to periodically delete old data from the local hard drives. At the moment data will be deleted 30 days after creation or modification. Users must ensure their data is copied off the local hard drives to a server or other storage solution before that time, otherwise they will lose data. The policy is itemised below:
- Where possible, data should be saved directly to the LMCB servers, but see item 2 below
- You may use the microscope computer hard drive to save data temporarily but it will be deleted once it has been on the hard disc for 30 days. Situations in which you might want to use the local hard drive include sessions involving long-term time-lapse where the network connection to the server may be interrupted or if you are a user from outside the LMCB who does not have access to the LMCB network
- Any data saved locally must be saved in the UserData folder, which will be on either the C: or D: drive depending on the microscope. Please see the guide below
- LMCB users must transfer their data to their own folder on the LMCB network as soon as possible after imaging
- UCL or external users must transfer their data to a UCL share, DVD or a portable hard drive as soon as possible. All removable media must be scanned for malware and disinfected before being plugged into a microscope computer. Specific portable drives dedicated to image data are strongly recommended
- Users must book sufficient time on the machine to copy their data after acquisition (see Data Transfer Rates below)
- Users must delete their data from the local hard discs after it has been copied
- It may be necessary to delete data and carry out maintenance out of schedule, in which case users will be alerted either via the Faces Scheduling System or the lmcb-lm-users email
This policy may be revised in future if the rate at which data is generated increases to such a level that local hard discs must be emptied more frequently than once every 30 days
Data Transfer Rates
I have transferred test data sets to the server and USB drives to estimate the amount of time required to copy data. The results are below. Bandwidth on the network or computer could change from time to time so I can't guarantee you will always get these speeds. You may even be able to achieve faster speeds.
Transfer of 103 GB via Gigabit Ethernet from the Vox to confocalstore2 on Friday 25th April took 24 minutes.
Transfer of 103 GB via USB 2.0 from Vox to an NTFS formatted portable hard disc on Friday 25th April took 61 minutes.
This PDF explains how to calculate how big your data set will be.