15-MQN Analysis of Measurement Instruments

Classical Test Theory and Item Response Models

The course gives a broad introduction to the field of psychometrics, followed by a number of advanced topics which give an impression of current developments.
The introduction will cover classical test theory, including generalizability theory and item response theory (IRT). The practical of generalizability theory pertains to analyzing data with multiple raters and multiple tasks. As applications of IRT, the topics of test equating and differential item functioning will be treated. The students will learn to use standard state of the art user software for these analysis, the public domain package MIRT.
The advanced topics are multidimensional IRT and multilevel IRT, such as used in large-scale educational surveys such as PISA, TIMSS and PIRLS and in school effectiveness research. These topics will be addressed in a Bayesian framework, and students will learn to build applications to analyze these IRT models using Markov chain Monte Carlo methods. In the lab meetings JAGS and R will be used to analyze item response data using Bayesian inference.

Course objectives:
  • Understanding of classical and modern test theory and the relation between the two. 
  • Understanding of applications of test theory, such as reliability, generalizability and validity, item analysis, optimal test construction, differential item functioning; test equating, incomplete test administration designs, and multilevel analysis of large scale educational surveys and school effectiveness research. 
  • Experience with generalizability theory analyses and software 
  • Experience with IRT analyses and software for item analysis 
  • Experience with Bayesian IRT analyses and software for survey and effectiveness research 
Requirements/entry level:
Knowledge of statistics and methodology in the field of the behavioral and social sciences
Knowledge of educational research

Prof. Dr. Cees Glas (c.a.w.glas@utwente.nl)
Dr. Hanneke Geerlings (h.geerlings@utwente.nl)

Lecture 1 Introduction CCT and IRT
Subjects Overview of classical test theory, generalizability theory and item response theory, the relation between the various approaches and some important applications such as evaluation of reliability and validity, item analysis, and optimal test construction
Practical 1 Generalizability analyses using SPSS

Lecture 2 Applications of Item Response Theory
Subjects Applications treated are differential item functioning; test equating, incomplete test administration designs, missing data, multidimensional IRT models, i.e., full information factor analysis models.
Practical 2 Detection of differential item functioning and test equating with MIRT

Lecture 3 Bayesian Item Response Theory
Subjects The Bayesian IRT framework will be outlined, and a number of applications will be treated, including structural linear IRT models, such as ANOVA and latent regression models.
Practical 3 Bayesian estimation of an IRT model enhanced with a generalizability theory model as a structural model.

Lecture 4 Multilevel Item Response Theory
Subjects The principles of basic multilevel modeling will be reiterated and then brought into the framework of Bayesian IRT. A number of applications will be treated, including structural linear IRT models as used in PISA, TIMSS and PIRLS.
Practical 4 Bayesian estimation of an IRT model enhanced with a multilevel structural model.

Specification of the workload:
Sessions: 4 x 7 hours = 28 hours.
Preparation: 4 x 6 hours = 24 hours.
Writing report of practical: 4 x 8 hours = 32 hours.
84 hours in total

February 5-6, 2015
February 12-13, 2015
Vergadercentrum Vredenburg, Utrecht

Maximum number of participants:24

For all 4 practicals includes a personal assignment. Following the assignment, the relevant output and a report must be filed. Students will receive feedback.

Required reading (during the course before sessions):
  • Brennan, R. L. (1992). Generalizability theory. NCME Instructional Module 14. Public Domain.
  • Glas, C. A. W. (2012). Psychometric theory. Reader, University of Twente.
  • Glas, C. A. W. (2010). The MIRT package. Software, Reader, University of Twente.
  • Holman, R., & Glas, C. A. W. (2005). Modeling non-ignorable missing data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology. 58, 1-17.
  • Kim, S.-H. (2001). An evaluation of a Markov Chain Monte Carlo method for the Rasch model. Applied Psychological Measurement, 25, 163-176.
  • Korobko, O.B., Glas, C.A.W., Bosker, R.J. & Luyten, J.W. (2008). Comparing the difficulty of examination subjects with item response theory. Journal of Educational Measurement, 45, 139–157.
Recommended reading, to become more familiar with the subject:
ICO Education,
15 May 2014, 00:11