ISLS General Announcements

Keep up to date with all the latest ISLS news!

A Focus on Content: The Use of Rubrics in Peer Review to Guide Students and Instructors

Congratulations to Dr. Ilya M. Goldin on his successful completion of his dissertation entitled:  A Focus on Content: The Use of Rubrics in Peer Review to Guide Students and Instructors (2011)

Students who are solving open-ended problems would benefit from formative assessment, i.e., from receiving helpful feedback and from having an instructor who is informed about their level of performance. Open-ended problems challenge existing assessment techniques. For example, such problems may have reasonable alternative solutions, or conflicting objectives. Analyses of open-ended problems are often presented as free-form text since they require arguments and justifications for one solution over others, and students may differ in how they frame the problems according to their knowledge, beliefs and attitudes. This dissertation investigates how peer review may be used for formative assessment. Computer-Supported Peer Review in Education, a technology whose use is growing, has been shown to provide accurate summative assessment of student work, and peer feedback can indeed be helpful to students. A peer review process depends on the rubric that students use to assess and give feedback to each other. However, it is unclear how a rubric should be structured to produce feedback that is helpful to the student and at the same time to yield information that could be summarized for the instructor. The dissertation reports a study in which students wrote individual analyses of an open-ended legal problem, and then exchanged feedback using Comrade, a web application for peer review. The study compared two conditions: some students used a rubric that was relevant to legal argument in general (the domain-relevant rubric), while others used a rubric that addressed the conceptual issues embedded in the open-ended problem (the problem-specific rubric). While both rubric types yield peer ratings of student work that approximate the instructor’s scores, ratings elicited by the domain-relevant rubric was redundant across its dimensions. On the contrary, peer ratings elicited by the problem-specific rubric distinguished among its dimensions. Hierarchical Bayesian models showed that ratings from both rubrics can be fit by pooling information across students, but only problem-specific ratings are fit better given information about distinct rubric dimensions.

Professional Note:
Ilya is a Consulting Principal Investigator and Research Scientist for
several companies working to advance education and public health. His
dissertation and other materials are available at

Ilya Goldin

University of Pittsburgh
Advisor: Kevin D. Ashley, JD, Ph.D.

Joint EARLI SIG 20 & SIG 26 meeting: August 22-24,...

Member Log In

Please note that we have updated our website & all prior ISLS members must create a new profile

For information regarding membership renewal or changing membership plans click here


Please click here to become a member


Job Announcements

View positions in the Learning Sciences.

Programs of Study

Read about programs of study in the Learning Sciences.

Submit News Topics:

To submit a news item, please send an email with the subject of "ISLS NEWS" to the ISLS with the following information:

  1. Title of Announcement
  2. Start date, if any (please allow 1 week)
  3. Brief description (60 words or less)
  4. URL for further details and/or an email address for contact person