Data Storage and Backups

Light microscopes, especially those used for time-lapse microscopy or super-resolution imaging, can generate huge amounts of data. In order that the core microscopes continue to function it is necessary to periodically delete old data from the local hard drives. At the moment data will be usually be deleted 30 days after creation or modification. Users must ensure their data is copied off the local hard drives to a server or other storage solution before that time, otherwise they will lose data. The policy is itemised below:

• If you are an LMCB staff member you must save or copy data to your project folder on the UCL Active Directory. You can get a project folder by filling in the web form at this address: http://www.ucl.ac.uk/lmcb/storage-request-form. Ask your group leader first because there might already be a folder for your project. If you have any problems then please email local IT support: lmcb-it-help@ucl.ac.uk. 
• You may use the microscope computer hard drive to save data temporarily but it will be deleted once it has been on the hard disc for 30 days (or 14 days for the Vox, Elyra, NSTORM and FV1200). Situations in which you might want to use the local hard drive include sessions involving long-term time-lapse where the network connection to the server may be interrupted or if you are a user from outside the LMCB who does not have access to the LMCB network
• Any data saved locally must be saved in the UserData folder, which will be on either the C: or D: drive depending on the microscope. Please see the guide below
• You are responsible for copying your own data off the microscope computers
• LMCB users must transfer their data to their project folder on the Active Directory as soon as possible after imaging
• UCL or external users must transfer their data to a UCL share, DVD or a portable hard drive as soon as possible. All removable media must be scanned for malware and disinfected before being plugged into a microscope computer. Specific portable drives dedicated to image data are strongly recommended
• You must book sufficient time on the machine to copy your data after acquisition (see Data Transfer Rates below)
• You must delete your data from the local hard disc after it has been copied
• It may be necessary to delete data and carry out computer maintenance out of schedule, in which case you will be alerted either via the Faces Scheduling System or the lmcb-lm-users@ucl.ac.uk mailing list
• This policy may be revised in future if the rate at which data is generated increases to such a level that local hard discs must be emptied more frequently than once every 30 days

The table below lists those microscopes that have defined UserData folders. More will be added as time goes on.

Data Transfer Rates

I have transferred test data sets to the server and USB drives to estimate the amount of time required to copy data. The results are below. Bandwidth on the network or computer could change from time to time so I can't guarantee you will always get these speeds. You may even be able to achieve faster speeds.

Transfer of 103 GB via Gigabit Ethernet from the Vox to confocalstore2 on Friday 25th April took 24 minutes.

Transfer of 103 GB via USB 2.0 from Vox to an NTFS formatted portable hard disc on Friday 25th April took 61 minutes.

This PDF explains how to calculate how big your data set will be.