Document a Dataset

DDI is a flexible standard and different users are at will to implement the standard in slightly different ways. This is a strength, but when moving between different implementations some adjustments need to be made.

Data files should meet certain criteria, this generates a standard SledgeHammer output, which is then lightly edited to provide a consistent structure for ingest into Colectica Repository.

Variable metadata workflow

Data Files

Studies should be encouraged to generate data files that are of the standards the UK Data Service

• use meaningful and self-explanatory variable names, codes and abbreviations
• variable and value labels must be clear and consistent, avoiding truncation of variable and value labels
• non-compliant characters, such as &, @ and <>, should be removed
• ensure no repetition of variables, especially redundancy in derived variables
• ensure consistent treatment and labelling of missing values
• extraneous information such Document entries should be stripped out

The workflow see [Building the Repository Guide], expects a simple mapping of dataset to questionnaire. The naming of the datafile should be consistent with that of the questionnaire it is collected from.

Data files should ideally be in SPSS format. Some guidance on SPSS file preparation is given below.

Use of SledgeHammer

SledgeHammer is a product released by Metadata Technology (North America) and allows the extraction of metadata from a wide range of data formats. Although it can be run interactively, the project uses batch files to allow a consistent generation of output

The project uses a restricted set of these commands

Batch File

Each dataset should have a batch or command file which calls the sledgehammer-cl.bat file, and lists the options above An example is:

sledgehammer-cl.bat" ^
-ag uk.cls.bcs70        ^
-rename bcs_75_mcs ^
-ddi 3.2-RP               ^
-ddipd proprietary ^
-har                ^
-ddilang en-GB        ^
-ddiref urn   ^
-ddiurn canonical ^
-pretty      ^
-opt full ^
-scan ^
-stats max,min,mean,mode,valid,invalid,freq,stdev ^
../bcs70/bcs_1975/bcs_1975_masc.sav

Metadata Edits

For display purposes and for ease of navigation and ingest, a consistent set of names should be applied to the output from SledgeHammer prior to ingest through a series of edit scripts. These are written in python, and if they cannot be run at the study, can be run at CLOSER prior to ingest. Short Name - is the name of the metadata bundle with which the dataset of associated with Long Name - the name you want to display as a human readable description DOI - if available, this allows the user to navigate to the DOI and relevant citation and is provided for the user Public - 1 is public 0 is not public An example of this file is shown below

This should be a tab delimited file called rename_list.txt

Control File

A control file can be used to batch up the batch files and then run the edits across all the files:

call pms.bat
call ncds8_sc.bat
call mcs_03_na.bat
call bcs_1970.bat
call mcs4_teacher.bat
call mcs5_teacher.bat
call mcs5_sc.bat
call bcs_75_mcs.bat
python g://db//bin//fandr.py
python g://db//bin//fandr2.py
python g://db//bin//fandr3.py
python g://db//bin//fandr4.py
python g://db//bin//fandr5.py
python g://db//bin//fandr6.py
python g://db//bin//fandr7.py
python g://db//bin//fandr8.py
python g://db//bin//fandr9.py
python g://db//bin//fandr10.py

Outputs

For each dataset a DDI 3.2 file called [shortname].ddi32.rp.xml will be generated.

Checking

If the edits are run, the file can be imported into Colectica Designer to check that it is well formed.

SPSS File Preparation

SPSS will hold lots of hidden information, Sledgehammer wil try to use this and can lead to issues when outputting the DDI-L XML. We would recommend using something like this to get rid of this extraneous information This replaces a file label (often the location of the original file) with the bundle name, and to drop any document(s):

get file="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".
FILE LABEL "bcs_75_msc".
DROP DOCUMENT.
EXECUTE.
sysfile info file="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".
save outfile="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".

1. Setup Sledgehammer

• Run the SledgeHammerInstaller.exe to install the software- usually on C: drive programs. Must have admin permissions.
• Run the software Sledgehammer.jar or from shortcut and select the license. Go to settings > Licensing > browse to the location of the license i.e. JonJohnson.UCL.ent20.lic

2. Setup Python

• Download version 2.7
• Add path for windows Run > sysdm.cpl > Advanced > Environment Variables. In the System variable window, find the Path variable and click Edit. Add the location of the installed Python file e.g. C:\Python27\. Test using python -v.
  
  

3. Setup DDI-Flavour

• Download DDI- Flavour from https://github.com/CLOSER-Cohorts/ddi-flavour

4. Install Java runtime

5. Prepare SPSS files

• Only run Sledgehammer on SPSS files.
• SPSS will hold lots of hidden information, Sledgehammer will try to use this and can lead to issues when outputting the DDI-L XML. We would recommend using something like this to get rid of this extraneous information This replaces a file label (often the location of the original file) with the bundle name, and to drop any document(s):

get file="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".
FILE LABEL "bcs_75_msc".
DROP DOCUMENT.
EXECUTE.
sysfile info file="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".
save outfile="G:\DB\closer_data\bcs70\bcs_1975\bcs_1975_masc.sav".

 6. Prepare batch files

• One .bat file per SPSS file e.g. ncds8.bat
• One .txt file for all files to run DDI-Flavour after Sledgehammer e.g. renaming_list.txt 
• One overall .bat file to run all the batch files and DDI-Flavour
• See https://closer-docs.readthedocs.io/en/latest/datasets.html

7. Run Sledgehammer and DDI-Flavour

• Run the batch file from the command line. Do not run the Sledgehammer from the GIU.
• e.g. run_files.bat
• The output will be in the directory of the .bat files.