18-MPE Program Evaluation in Education: Empirical Research Methods to Identify Causal Effects

Course coordinators:
1. Dr. Ilja Cornelisz (i.cornelisz@vu.nl)
2. Dr. Chris van Klaveren (c.p.b.j.van.klaveren@vu.nl)

Course description:
Estimating the causal effects of educational innovations are of great importance, because these estimates allow us to make predictions about the consequences of changing circumstances or policies. Several estimation methods have been developed to identify causal effects in educational science, such as RCTs, Difference-in-Difference designs, Regression-discontinuity approaches, natural experiments, multi-level approaches and statistical matching techniques. This course provides researcher with the knowledge and empirical skills necessary to identify whether educational programs are effective.

Course objectives:
The objectives of this course are that participants:
· Can distinguish between the available different (quasi-)experimental methods.
· Understands the strengths and weaknesses of these (quasi-)experimental methods.
· Can empirically estimate (quasi-)experimental regression models.
· Can evaluate if studies provide evidence and can be included in systematic literature reviews.

Requirements/entry level:
This course assumes that PhD-students have:
· Basic knowledge in statistics
· Basic knowledge in bivariate correlation and regression analysis
· Interest and motivation in data analysis and evaluation methods

Course Programme
The course consists of ten meetings divided over four days. Before the first meeting students are required to read a scientific article on how to perform systematic literature review (Van Klaveren, C. & De Wolf, I. ,2015). This article together with the students’ own research idea/proposal will be the point of departure of the course.

Meetings 2 to 7 discuss the different available empirical designs and each meeting consists of
1. An interactive lectures (with use of student response tool mentimeter), and
2. A lab session in which students replicate the results of existing scientific articles.

On the start of the first course day students receive a take-home assessment which should be handed in before day 4. Meeting 8 is a plenary discussion of the take-home assessment. The final two meetings are group discussions in which we discuss ‘whether there is a ranking with regards to what method works best’ and ‘what students have learned with respect to their own planned research’.

Meetings:
Day 1
Meeting 1 - Systematic Reviews in Education Research
(2 hours Lecture)
Application to own Research
(1 hour lecture)
Meeting 2 - RCT & Natural Experiments
(2 hours lecture & 2 hours lab session)

Day 2
Meeting 3 - First difference, Difference-in-Differences & Regression Discontinuity
(2 hours lecture & 2 hours lab session)
Meeting 4 - Instrumental Variable (IV) Analysis Part 1
(2 hours lecture & 1 hours lab session)

Day 3
Meeting 5 – IV analysis Part II
(2 hours lecture & 1 hours lab session)
Meeting 6 Panel data
(2 hours lecture & 1 hours lab session)
Meeting 7 - Statistical Matching Approaches
(1 hour lecture)

Day 4
Meeting 8 – Discussion of Program Evaluation Assessment
(Discussion of assessment, 3 hours)
Meeting 9 – Systematic Reviews in Education Research: What method works best?
(2 hours group discussion)
Meeting 10 – Estimation Designs and Own Research: What have we learned?
(1 hour group discussion)

Instructional Format
The meetings offers a combination of lectures,lab sessions and group discussions:
· Lectures (with use of student response tool mentimeter)
· Lab sessions in which students empirically estimate the various designs with software demonstration (SPSS) on how to estimate the various evaluation designs empirically.
· Group discussion in which students

Dates and locations:

15 februari

Vredenburg 19 - Utrecht

8 maart

Studiecentrum OU Utrecht - Room  7

9 maart

Studiecentrum OU Utrecht - Room  7

15 maart

Vredenburg 19 - Utrecht