Examining The Impact of Calibrated Peer Review (CPR) on Student Writing Developed Through Web-Based Ecological Inquiry Projects

E-learning tools such as Calibrated Peer Review (CPR) have made writing assignments easier to implement and grade; however, we have limited knowledge of how CPR affects student scientific writing. Past CPR research has examined how CPR generated scores change across multiple CPR writing assignments for the purpose of reporting student learning gains. This study will not rely on CPR generated score data. This study (1) independently evaluated the impact of CPR on student writing of ecological inquiry report components using a grading criteria instrument and (2) explored how the revision process influenced the quality of ecological inquiry report components through text analysis.

A web-based science inquiry project was implemented in a large (up to 500 students) introductory ecology course. Students observed grizzly bears at McNeil River Falls in Alaska using Bear Cam picture stills. They developed and tested hypotheses about grizzly bear spatial distribution and interactions and reported findings in individual ecological inquiry reports. Students submitted reports to CPR and anonymously reviewed three peer reports and self-assessed their own. Finally, students were given one-week following CPR to revise reports based on peer reviews and submit online.

A 28-item grading criteria instrument (9 scales) was used to examine how students revised ecological inquiry reports post CPR. Eight paired t-tests were used to assess the pre-post CPR changes in scores for individual grading criteria scales or components. Cohen's d effect size was used to explore how achievement or performance level, ethnicity, gender and major influenced student text changes to ecological inquiry report components post CPR. Text analysis using a subset of 27 sample reports (pre-post CPR) assessed the amount and location of text changes and the impact of these revisions on the quality of ecological inquiry report components. Common errors in ecological inquiry report components post CPR were also analyzed.

Results showed that CPR and revision significantly improved the scores related to the objective, sampling and discussion scales. Analyses using Cohen's d effect sizes illustrated interesting but inconsistent patterns related to the influence of student performance level, gender, ethnicity, and major on pre-post CPR score gains. Text analysis revealed the majority of helpful revisions were related to making the objective identifiable, reporting of sample size and discussion of study limitations and future questions raised by individual ecological inquiry projects. Text analysis shows three common reasons participants failed to meet grading criteria post CPR. Un-testable hypotheses, insufficient descriptions for sample selection, data analysis, variables collected and revisions of only easy grading criteria components. This study provided direct evidence of CPR's effects on student writing and provided a greater understanding of pattern of revision process following CPR.