15-T7 Research synthesis including meta-analysis

Course description:
Synthesising results from previous studies is a vital step in educational science. What have previous studies for a specific topic found as results? Do these findings converge or not? And can differences in findings across studies be explained by theoretically inherent issues or by features of the research design employed? This course addresses issues related to systematic reviews and meta-analysis and helps participants to develop knowledge and skills to both better understand and evaluate syntheses written by others and do their own synthesis.

Course objectives:
In this ICO course we will focus on the following objectives:
  • Formulating the question that is central to the synthesis 
  • Finding studies that can be used in the synthesis, and setting inclusion/exclusion criteria 
  • Scoring studies on various quantitative and qualitative characteristics 
  • How to perform a qualitative analysis of these studies How the do a meta-analysis (calculating effect sizes, constructing study weights, predicting differences between study outcomes, how to deal with publication bias, etc.) 
Requirements/entry level: 
We assume that all participants have a basic knowledge of statistics (correlation, regression, etc.) and are able to write an overview of scientific literature.

Course coordinators:
1. Dr. A.C. (Anneke) Timmermans (a.c.timmermans@rug.nl)
2. Dr. H. (Hanke) Korpershoek (h.korpershoek@rug.nl
3. Dr. D. (Danny) Kostons (d.d.n.m.kostons@rug.nl)
4. J.M. (Marjan) Faber, MSc. (j.m.faber@utwente.nl)

Day 1:

During the first day we will give a brief overview of the rationale behind the knowledge building process, starting from small scale qualitative studies, to correlational/observational studies, quasi-experiments, and randomized controlled trials to synthesising results. Emphasis will be given on why one would like to conduct a systematic review or meta-analysis and possible research questions.
An example of a meta-analysis will be presented by Marjan Faber, a PhD-student from the ICO Theme group Schools and the societal context of education. In this presentation there will be attention on the steps taken to conduct a meta-analyses, points of discussion encountered while performing the meta-analyses, the results and discussion.
During this first day participants will form groups of two or three students for the assignment of the course and start thinking about a topic for a small review study that will be conducted between the second and third day of the course.

Day 2:
During the second day we will look more closely into the process of reviewing, including 1) a systematic search of literature, 2) criteria for in- or exclusion of studies, and 3) coding schemes and coding the studies that meet the in- or exclusion criteria. Besides methodological issues related to these aspects of reviewing, the steps will be illustrated by means of two recent review studies (Korpershoek, Harms, de Boer, van Kuijk, & Doolaard, 2014; Kostons, Donker-Bergstra, & Opdenakker, 2014). In between lectures and examples the groups of students will start working on the task for this course related to the three topics mentioned above.

In between the second and third day the groups of students will work on the task (a specification can be found below).

Day 3:
The third day of the masterclass will start with presentations of the groups of students on their experiences and results of the review task. Afterwards there is time for discussion and feedback.
Thereafter we will start with an in-depth explanation of meta-analyses, summarizing results of multiple studies in a quantitative way. During this day we will focus on the necessary statistical information reported in studies, measures of effect size, transformations of effect sizes, precision measures, weighing evidence, summarizing effect sizes over studies (and forest plot), publication bias(funnel plot).

Day 4:
The fourth day will start with a hands-on session working with the software Comprehensive Meta-analysis. Participants are asked to bring their own laptops and install the 10-day trial version of Comprehensive Meta-analysis on their laptops prior to the fourth course day. The remaining content on the fourth day is partly based on the interest of the participants. Based on the interest of the participants we will offer the opportunity to split into smaller groups. Possibilities are:
- Advanced topics in meta-analyse such as the distinction between fixed and random effects models and meta-regression
- Further discussion on other methods of synthesising results of multiple studies.

At the end of the last day we will organize a group discussion on the value and critique of meta-analyses and other forms of summarizing results.

Specification of the workload:
The course requires a total investment of 84 hours.
- 30 hours will be attending meetings and doing hands-on analyses,
- 20 hours for studying texts (see Literature)
- 34 hours for working on the assignment (conducting several steps from a systematic synthesis and preparing a presentation).

Meetings are planned between 10.30 and 16.30 hours.
Day 1: 15 October 2015
Day 2: 16 October 2015
Day 3: 19 November 2015
Day 4: 20 November 2015


The meetings on October 15th and 16th and November 20th are organised at the Utrecht Location of the Open University. The meeting on November 19th will be organised in Vredenburg conference centre Utrecht.

Maximum number of participants: Max 15 participants.

Participants complete the course when they attend all meetings, actively contribute to the meetings, prepare the meetings according to the guidelines, and complete the assignment.

Required reading before day 1 and 2:
· Boxem, A.A.K., Becht, H., van Bentum, M., Braaksma, H., Drent, M., Vroom-van Gorcum, J., & Wuite-Harmsma, H. (2014). Methodisch en Efficiënt Wetenschappelijke Informatie Zoeken. Enschede: Universiteit Twente. (Selection)
· Petticrew, M. & Roberts, H. (2006). Systematic Reviews in the Social Sciences. A practical guide. Malden, MA: Blackwell Publishing. (Selected chapter, pdf available via dropbox)
Example of best evidence synthesis:
· Slavin, R., Lake, C., Davis, S., and Madden, N. (2011), Effective Programs for Struggling Readers: A Best-evidence Synthesis. Educational Research Review 6, 1–26.

Required reading before day 3 and 4:
· Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, Calif: Sage Publications. (Selected chapters)
Example of meta-analyses
· Patall, E. A., Cooper, H., Robinson, J. C. (2008). Parent Involvement in Homework: A Research Synthesis. Review of Educational Research,78,: 1039-1101
Example of meta-analysis and accompanying methodological discussion:
· Kingston, N., & Nash, B. (2011). Formative assessment: A meta-analysis and a call for research. Educational Measurement: Issues and Practice, 30, 28 – 37.
· Briggs, D. C., Aracelli Ruiz-Primo, M., Furtak, E., Shepard, L., & Yin, Y. (2012). Meta-analytic methodology and inferences about the efficacy of formative assessment. Educational Measurement: Issues and Practice, 31, 13 – 17.

Optional reading:
Example of meta-analysis:
· Korpershoek, H., Harms, G. J., De Boer, H., Van Kuijk, M. F., & Doolaard, S. (2014). Effective classroom management strategies and classroom management programs for educational practice. A meta-analysis of the effects of classroom management strategies and classroom management programs on students' academic, behavioural, emotional, and motivational outcomes. Groningen: GION Onderwijs/Onderzoek
Example of meta-regression:
· de Boer, H., Donker, A. S., & van der Werf, M. P. C. (2014). Effects of the Attributes of Educational Interventions on Students’ Academic Performance: A Meta-Analysis. Review of Educational Research, 84(4), 509-545. 10.3102/0034654314540006

ICO Education,
18 May 2015, 04:10