A Focus on Content: The Use of Rubrics in Peer Review to Guide Students and Instructors

Congratulations to Dr. Ilya M. Goldin on his successful completion of his dissertation entitled: A Focus on Content: The Use of Rubrics in Peer Review to Guide Students and Instructors (2011)

Abstract: Students who are solving open-ended problems would benefit from formative assessment, i.e., from receiving helpful feedback and from having an instructor who is informed about their level of performance. Open-ended problems challenge existing assessment techniques. For example, such problems may have reasonable alternative solutions, or conflicting objectives. Analyses of open-ended problems are often presented as free-form text since they require arguments and justifications for one solution over others, and students may differ in how they frame the problems according to their knowledge, beliefs and attitudes. This dissertation investigates how peer review may be used for formative assessment. Computer-Supported Peer Review in Education, a technology whose use is growing, has been shown to provide accurate summative assessment of student work, and peer feedback can indeed be helpful to students. A peer review process depends on the rubric that students use to assess and give feedback to each other. However, it is unclear how a rubric should be structured to produce feedback that is helpful to the student and at the same time to yield information that could be summarized for the instructor. The dissertation reports a study in which students wrote individual analyses of an open-ended legal problem, and then exchanged feedback using Comrade, a web application for peer review. The study compared two conditions: some students used a rubric that was relevant to legal argument in general (the domain-relevant rubric), while others used a rubric that addressed the conceptual issues embedded in the open-ended problem (the problem-specific rubric). While both rubric types yield peer ratings of student work that approximate the instructor’s scores, ratings elicited by the domain-relevant rubric was redundant across its dimensions. On the contrary, peer ratings elicited by the problem-specific rubric distinguished among its dimensions. Hierarchical Bayesian models showed that ratings from both rubrics can be fit by pooling information across students, but only problem-specific ratings are fit better given information about distinct rubric dimensions.

Professional Note: Ilya is a Consulting Principal Investigator and Research Scientist for several companies working to advance education and public health. His dissertation and other materials are available at http://ilyagoldin.com.

Ilya Goldin, University of Pittsburgh
Advisor: Kevin D. Ashley, JD, Ph.D.

Related Articles

ISLS 2024 Reviewer Invitations

The ISLS 2024 team is thrilled to have received a large number of papers, posters, and symposia for this year’s annual meeting, a total of 893 submissions! We have currently sent out reviewer invitations to prior reviewers and 2024 submitters. If you fall under either of those categories and have NOT received an invitation and would like to volunteer, please check your email and spam folders.

JLS Outstanding Paper (2022): Utilizing dance resources for learning and engagement in STEM

This paper authored by Folashadé Solomon, Dionne Champion, Mariah Steele and Tracey Wright received the Outstanding Paper Award from the Journal of the Learning Sciences. As the selection panel comments, “By employing culturally responsive pedagogy, the authors established a connection between the learning of physics and dance education, thereby promoting access and equity…The meticulous analysis provided insights into how dance, as an embodied form of knowledge, facilitated a transformation in the black girls’ relationship with physics.”