Processing PUMF Files
This note provides a vignette showing how the Statistics Canada public use microdata file (PUMF) for the Survey of Financial Security 2016 can be converted into a usable format for processing in R to produce weighted cross tabulations and charts..
The key package used in this vignette is the MEMISC package developed by Martin Elff at the Zeppelin University in Friedrichshafen Germany where he teaches political science. His package provides an infrastructure for the management of survey data including value labels, definable missing values, recoding of variables, production of code books, and import of (subsets of) 'SPSS' and 'Stata' files is provided. Further, the package allows to produce tables and data frames of arbitrary descriptive statistics and (almost) publication-ready tables of regression model estimates, which can be exported to 'LaTeX' and HTML
Dr. Elff’s website contains a detailed discussion about how to use the package for analyzing microdata with an emphasis on categorical variables. Managing items in a survey data set with value labels and recoding them is one of its many functions. However, of particular importance to this vignette is the infrastructure that he supplies to process raw tax files and variable and value labels defined in the SPSS format.
In the example discussed below, we will utilize the SFS2016 released in the summer of 2018. A code listing and log are included in the accompanying PDF.
We will also use the ggplot2 package from the tidyverse system to produce plots based on weighted cross tabulations of the data. One of the characteristics of the Statistics Canada pumf files is that they are weighted surveys. Although not done in this example, descriptive and other analysis using weights can be done with many other packages including HMISC and descr.
Converting the Data
The first stage is always to process the flat text files provided by Statistics Canada in the PUMF distribution. There is always a data file and control files for several packages including SPSS. In this example, we will use the SPSS version of the control files.
In this vignette, the conversion process has been done in a separate script from the analysis function. In this initial processing script, the first stage is to load the packages and define the location of the text file.
#minimum version should be memisc_0.99.14.12
#define locations of input spss files
print(paste("Current directory is",getwd()))
The next stage is to create a definition of the dataset using the spss.fixed.file import procedure saved in the variable sfs2016_efam.
The next stage is to use the dataset definition, the importer object, to create a dataset in R’s memory, write a codebook listing the variables. Value labels were not supplied for all variables in the dataset but were supplied for the weight variable. Because of the presence of these value labels, MEMISC considers the weight variable (pweight) to be a nominal categorical variable. The measurement of the variable is changed to “ratio” so that MEMISC will consider it as a numerical variable. The modified memory object (sfs2016_efam.ds) is then saved in an R save file for easy reloading in other R scripts. Other data edits cold be made before this save as well. However, in this example, such edits will be made in the analytical part of the example.
#modify the type for pweight because it is initial set to be a nominal
# categorical variable because value labels are supplied
Doing a codebook analysis for the pweight variable now shows that it is regarded as being numerical.
sfs2016_efam.ds$pweight 'Survey weights - PUMF'
Storage mode: double
Missing values: 9999999.9996-9999999.9999
Values and labels N Percent
9999999.9996 M 'Valid skip' 0 0.0
9999999.9997 M 'Don't know' 0 0.0
9999999.9998 M 'Refusal' 0 0.0
9999999.9999 M 'Not stated' 0 0.0
(unlab.vld.) 12428 100.0 100.0
The results from this script are shown in the PDFs. Any problems are likely result of incorrectly formatted SPSS control cards. The data file as delivered was longer than the record definition. This required some modifications to MEMISC. The minimum version for this example to work is memisc_0.99.14.12.
Some Basic Analysis
In the example script, we will define a new age variable grouping the survey respondents by birth year and then analyze some aspects of their mortgages. The value labels for several of the variables used will have to be supplied because these were omitted in the Statistics Canada deliverables.
The first stage sets up the directory and loads the saved data set from the previous script.
This creates a second copy of the DS object for additional modifications.
#loading tidyverse with mask some similarly named functions in memisc
print(paste("Current directory is",getwd()))
It is this copy (sfs2016_2.ds) that will be modified and saved in this script. Value labels will be added for several variables because they were omitted from Statistics Canada’s package. The definition of the value labels is found in the PUMF documentation provided by Statistics Canada.
The first stage is to create new age variable grouping the respondents into Boomers (born up to and including 1965), Generation Xers (born between 1966 and 1979 and the Millennial group born after 1979. The cut command is used to break a birth year variable into groups (1-3) on the basis of the break years. The latter are treated as closed to the right by default. Appropriate labels are provided.
#now to look at age
#now for the classification
#right=TRUE is default for cut - means groups are closed on the right
The variable is defined as a factor or ordered categorical variable.
The next set of code adds the grouped_birth_year as the variable age_cohort to the data set and supplies a description. Value labels are added for the type of mortgage interest and the term of the mortgage because these were omitted from the supplied definitions by Statistics Canada.
#now define some sample variables
#now start some variable edits in the sfs2016_2.ds object
description(age_cohort)<-paste(description(pagemie),"By Birth Year")
description(own_mortgaged)<-"Own home but have mortgage"
#we have to supply labels for a number of variables because Statcan's label file was incomplete
#we could subtract 2016 to show the number of years
description(years_to_renewal)<-"Year of Renewal from 2016"
#save the calculations so we don't have to recreate the variables we modified
Now we can start the analysis. The basic approach in this simple example is to use the xtabs function to produce weighted cross tabulations. By default, not-available (NA) observations are not processed.
The first cross tabulation looks at tenure of dwelling.
#now do a weighted frequency counts
#convert to a data frame for ggplot
labs(title="Economic Family Housing Tenure - 2016",
fill="Tenure",y="Economic Families (000)",x="Birth Year Cohort",caption="Source: SFS2016, JCI")
The data are in economic family units so the resulting xtab object is sc
The appendices show the scripts and logs. Note that comments may wrap because of formatting.