Chapter 11 Reproducibility and Documentation
Often times we treat an analysis like it is a one of a kind. This “job shop” mentallity is bad when it comes to institutional research. Often if a question is asked once it will probably be asked again. Most of the work in higher education is temporal. A question may not be relevent today, or give one results, but the same question asked a year later may generate a different answer. As such it is good practice to treat every question like it will be asked again. Designing processes to support this kind of approach may take some effort on the front end, as you have to treat each analysis like it will go into production, but the results in the end are much better. Everything you produce will be fit for distribution (as far as structure, documentation, not necessaryily FERPA and other ethical considerations) and easy to duplicate should the data change or a new consideration need be made.
The workflow described in chapter 20 is in support of reproducibility and repeatability. I use both of these words and I think that I should define them in the contect of research.
Reproducibility– Anyone who uses the same approach and code with new data should get similar outputs (within the confidence intervals specified). The key here is with new data.
Repeatability– You can run your analysis on the same data and get the same results each time.
Every analysis should strive to meet these two requirements. You can do this by following the approachs detailed in the workflow section of this book.
11.1 A Common Reporting Structure
An additional way to faciliate reproduvibility and repeatavility is through having a common reporting structure. Even if this structure is not the final document that is presented to the college president or the provost, it is a good practice to write in a common approach. The aim is two fold. First it forces you to ensure that you answer and document all of the common questions of the analysis. Secondly it allows you to instantly refers others to sections in the report. IF the reporting structure is common then if someone asks a question you can refer them to the “methods” section or please look in the background section to see if that was discussed or considered.
Consistently in presentation and approach also lends itself to a higher image. People become accustomed to a format, they know what to expect and they can spend more time digesting the analysis and less time on trying to navigate the format.
So what should the structure look like? The following proposal borrows heavily from academic journal articles but with a more action oriented approach. While the needs and preferences of institution may vary, some potential options include the following.
11.1.1 Executive Summary
What does the analysis suggest and what do you recommend based on it? Describe what the discovery was, what it indicates and suggest the next steps. The difference from an abstract in the academic journal is that youur audience is already acquainted with the research question. They don’t need you to rehash is. Be brief, clear and succinct. IF the president of the university only read 3 sentences of your report what want them to be? That is what you should find here.
This is the time where you can provide the narrative and set up the context for the analysis. It is in the backhround that you can introduce the topic that you are studying, what brought the question into exist. Also include what is the mission of the study. Is it to inform policies or change the direction of a program. This is critical to the background of the study. Remember you are writing here for your future self and for those who have no context for the study. For the key decision makers you have provided the executive summary. This section is to give all of the information necessary to understand the context.
This is also a place to indicate any previous studies or literature on the topic. This is not a formal literature review, but it should be a primer for what outside information was available and if there were previous studies done and what the findings were. This helps to provide some additional context as to why the new study exists or how it fits into existing literature.
This section is the meat of the report. In this section the entire method, method of data aquistion how the data was acquired is needed. Here is details about the statistical methods used should be detailed including checks of normality, verification of model fit applied and any explanation of new analytic approaches. This is certainly the most dense section of the report, but it is important to delineate these topics so it is clear what was done, what assumptions were made, and if there is sufficient background. Speak simply about complex things and aim to communicate in as simple language as possible.
This is the section in which to put all of the results. No interpretation is needed at this point, just the figures that were generated from the analysis. This last point helps when it comes to defending a position. If you make a clear delineation between your results and your interpretation others will feel that your analysis is more objective. Additionaly, it allows you to separate the argument from the interpretation and refocus any issues back to the results which are free of subjectivity. The argument can be made that the how you assembly the tables and figures introduces decisions, but these decisions and what is represented should be described in the method section. Thus you have telegraphed the analysis and depiction of the results.
Now for the conclusions and interpretation of the results. What did you learn from the study and how do the results support those conclusions. Does it answer the research question posed? Are there weaknesses in power, effect size, posterior probability?
This is another section that differs from purely academic works. Recommendations are the recommendations for the next steps. Not every report will have this section, but if you were asked to pose a question as to the effectiveness of a program, you can make your claim here and recommend actions based on the analysis. This should be a natural extension of the conclusions, but just calling out the actions required next. Often this section will just contain a few bullets that underscore the concrete actions or decisions that are required next.
If a senior leader in the company reads the executive summary and the recommendations they should understand the ressults of the study, what it means for the organization and what are the proposed next steps that they need to take.
11.1.7 What is it good for?
A thorough report is a nice way to document studies. As with all things you become a stranger to your own work as time passes. Having your analysis written out in a formal and rigourous way, while time consuming is a gift to your future self. You will probably re-do the analysis again with new data. New data become available and the research question that generated the report will probably come back to life (given the new data are our findings consistent with the old data? Did anything change?). Having a formal, repeatable report will help you answer these questions in a quick way.
Having a formal documented report also facilitates sharing of knowledge. Often times you cannot forsee how one analysis will become relevant in another context. With a well documented report you can share the report in the next context. This expedites knowledge sharing and organizational learning. Additionally, a repository can be started to house all the reports done. Senior leaders and others can browse these studies to see if a study has already been done and not need to commmision a new analysis. Or better yet in the presence of a report, come up with a better question. All of these are incredibly powerful.
11.2 Short Form Report
Unfortunately, sometimes we do not have time to document our analysis completely. Or we need to submit more of a brief. I still recommend writing the full report if not for others for yourself. As I mentioned your work becomes strange to you over time and you never know when you will need a formal write up. However, it is often a good practice to develop a one page report that distills the entire analysis to a single page. This is the big idea sheet.