13-06 Multiple Perspectives on Assessment

Research theme: 6 Assessment, Evaluation and Examination xxx
course description
This masterclass includes presentations and assignments which will encourage the participants to broaden their knowledge about assessment. Both conceptual and instrumental contributions will be included, to assure that participants understand that assessment is more than just implementing an instrument and has substantially wider educational implications.

course objectives
The course objectives of this masterclass are that participants:
  • are aware of the different purposes of assessment practices 
  • understand the use and consequences of assessment purposes and -instruments 
  • understand the value of (peer) feedback in assessment and underlying theories 
  • understand the basic principles of generalisibility theory and Item Response Theory 
  • understand the definition of assessment quality and underlying criteria 
  • understand the principles of portfolio assessment; 
  • develop a personal perspective on assessment 
  • are able to embed the course elements in the context of your own research; 
  • are able to write and present a position paper on assessment. 
To be admitted to the course, participants are required:
  • to motivate why they participate in this masterclass; 
  • to select a relevant article on assessment to be presented to the peer group. This article is published in English in a high impact journal, for example Learning and Instruction or Assessment in Education; 
  • to study several articles (to be announced several weeks before the course); 
  • to study the articles before each guest lecture. 
Entry requirements are:
  • interested in assessment, preferably conducting research on assessment; 
  • motivated to be engaged in discussions; 
  • basic assessment knowledge (see for example www.toetsenopschool.nl) 
Students who have specific questions to be addressed in the course are advised to send these to the course coordinators two weeks before the start of the course.
The course requires a total time investment of 100 hours, of which approximately 32 hours are for attending the meetings. Another 68 hours is planned for studying literature and the end assignment.

Prof. dr. Cees van der Vleuten (Maastricht University)
Dr. Dominique Sluijsmans (Zuyd Hogeschool and Maastricht University)
Prof. Dr. Jan van Tartwijk (Utrecht University)
Prof. dr. Theo Eggen (University of Twente, Cito)
Dr. Erik Driessen (Maastricht University)
Dr. Jan-Willem Strijbos (Ludwig-Maximilians-Universität München)
Dr. Desirée Joosten (Fontys University of Applied Sciences – teacher education and Open Universiteit)

The course encloses four contact days. In the first three days, a variety of assessment perspectives will be presented by lecturers who are all experts in the field of assessment. They will actively engage the participants through interactive discussions and assignments.
The fourth and final day of the course is reserved for presentation of the individual position papers and peer assessments.

Summary of the contributions:

Day 1.
Dr. Dominique Sluijsmans
Assessment for the future: Which elements drive learning on the long term?
To become successful lifelong professionals, learners need to develop expertise and engage in a variety of tasks that foster continuous self-improvement. In making the transition from university to work however, many starters experience difficulties in establishing and maintaining their professional learning. Moreover, current assessment practice in education requires learners to achieve standards with certification as the ultimate goal. Although it is widely acknowledged that continuous self-improvement is crucial in many professions (Boud, 2000), our current educational system is still dominated by assessment models that do not foster long-term learning. The fixed terms for standardised exams assume uniformity of learners and a finiteness of learning: for those who are among the ‘best’ of their peer group or attain the minimal standards, there is no reason to continue learning. Moreover, these models reinforce learners’ test behaviour - the books disappear in the drawer after the final exams (Frederiksen, 1984). This short-term focus on assessment hampers learning in the long-term, because certified students may not be equipped to continue learning in the ways needed for employment. In this contribution, it is argued that if we strive for learning on the long-term both assessment models and the current purposes of assessment are not sufficient.

Dr. Jan-Willem Strijbos (Ludwig-Maximilians-Universität München)
Understanding your understanding of my understanding: The many facets of providing and receiving peer feedback
With the increased application of peer assessment in higher education and the renewed interest in feedback, the element of peer feedback in the context of peer assessment is receiving more attention. In particular the interactive aspects of peer feedback (e.g., sender characteristics on the reception of peer feedback) are at present not well-understood. Also the type and format of peer feedback appears to differ for various recipients (e.g., personal characteristics, like knowledge level, gender and educational level). During this contribution you will first experience peer feedback (both providing and receiving), and this will be used to discuss what we know and critically reflect on guidelines for implementing peer feedback in a classroom or organization.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112.
Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.

Day 2.
Dr. Desirée Joosten-ten Brinke (Fontys University of Applied Sciences – teacher education/Open Universiteit)
Assessment is an important determinant for the quality of the learning processes and it is more than constructing a test and present it to the students. In this part of the course, we will discuss the quality of assessment on different levels during the assessment cycle from design and development to analyses and evaluation. How to design a valid and reliable assessment with objective, efficient and relevant items that suits the requirements of a balanced assessment programme and the assessment policy of your institute? There will be a link of assessment for the educational practices and assessment for the research practice. Because also for research counts that reliable outcomes starts with reliable and valid (assessment) instruments.

Baartman, L. K. J., Bastiaens, Th. J., Kirschner, P. A., & Van der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programmes. Studies in Educational Evaluation, 32, 153-177.

Prof. dr. Cees van der Vleuten (Maastricht University) 
Generalizability theory is one of three theories on the reliability of measures (the others being classical test theory and item response theory). Generalizability theory is versatile, particularly because it applies to complex measurement situations in which multiple factors contribute to the noise in the measurement. In this session we will review basic concepts of generalizability theory, how it might be used and how estimates of reliability indices may actually inform to improve your own measurement problem in your research. 

Shavelson, R. J. and N. M. Webb (1991). Generalizability theory: A primer. Newbury Park, CA, Sage Publications.

Day 3
Dr. Jan van Tartwijk (Utrecht University) and Dr. Erik Driessen (Maastricht University) 
Portfolios and learning from experience 
To learn form experiences, trainees have to (a) self-assess their task performance, (b) reflect on what went right or wrong, why and what was their own part in this, and (c) makes plans for improvement. The literature shows that the quality of self assessment is problematic. Portfolios can help trainees to improve the validity of their self assessment and ground their reflections. Portfolio can also be used as tools for summative assessment. For assessors the challenge is to combine evidence from various sources which is gathered in different contexts. Strategies from qualitative research can be helpful here. In our contribution, first we will present an overview of research findings about the quality of portfolio assessment for formative and summative purposes. Subsequently, the participants will practice with assessing and discussing competence using information from a portfolio. 

Prof. dr. Theo Eggen (University of Twente)
Item Response Theory
A basic introduction will be given in the theory and the applications of Item Response Theory (IRT) in educational settings. Attention will be paid to the differences between Classical Test Theory (CCT) and IRT. The characteristics and properties of the commonly used IRT models for dichotomously scored items will be treated. Working with IRT models in practice, that is in calibration, test construction and measurement will be illustrated. Finally a survey of available IRT models and computer programs will be presented.

Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), p 253-262.

Day 4
Presentations and peer-assessments.
Specific information will be announced several weeks before the course.

Participants are asked to prepare a two-page position paper and a presentation on two questions:
  1. How do you think that assessment will contribute to learning? 
  2. How did this masterclass contribute to your understanding of assessment? 
The contributions of the guest lecturers and the provided literature can be used as input for the position paper. Each participant has to put forward an argumentation on how they will incorporate the ideas from the course in their future work and/or research.

Each position paper will be peer-assessed by one member of the course group. The position paper will be presented on the final meeting to their peers and the course coordinators. Based on the quality of the presentation, the discussion, and quality of the peer assessment, a certificate will be handed out.

Each participant selects one article on assessment and presents this to the peers on the first day. This article is published in English in a high impact journal, for example Learning and Instruction or Assessment in Education. This will result in a first ‘body of knowledge’ on assessment. All articles are sent to the participants an have to be read before the course. The literature for the second and third day will be announced several weeks before the start of the course.

  • Wednesday February 27, 2013
  • Thursday February 28, 2013 
  • Friday March 1, 2013 
  • Wednesday, March 27, 2013 NEW DATE!!! (instead of March 15) 
Location : Maastricht University

Registration deadline: January 15, 2013

maximum number of participants:16