Developing Progress Monitoring Measures in Algebra Using Item Response Theory

As the emphasis on improving national academic standards and performance has grown increasingly focused on high-stakes testing in recent years, education professionals have begun exploring methods of measuring and tracking student improvement throughout the school year. Curriculum Based Measures (CBM) have been developed to assess academic growth across the core areas of the elementary curriculum (reading fluency, comprehension, writing, arithmetic, and calculation) and have been proven psychometrically sound. CBM are designed to be easily and quickly administered and scored, and to provide useful feedback to teachers regarding specific skill or content areas in which students may be progressing at less than optimal rates. Unfortunately, successful development of CBM for use in secondary education has been elusive. In particular, development of CBM for algebra has proven challenging, due in part to the quantity and variety of new skills that students are taught in a single year of instruction. To date, Project AAIMS (Algebra Assessment and Instruction – Meeting Standards) has produced three promising instruments to be used as algebra CBM. Of them, the Basic Skills probe is the focus of this study. Although reliability was easily established, field testing of the Basic Skills probe has heretofore failed to demonstrate adequate validity for it to be considered psychometrically sound.

In this study, item response theory (IRT) has been used to construct two new forms of the Basic Skills probe, using items from the original twelve forms as an item bank. The goal of this study was to apply IRT to these two new forms in order to determine their validity as instruments that assess algebra skills across a wide spectrum of performance levels. In addition, exploratory factor analysis was used to determine if more than one underlying factor is assessed by the instrument. Finally, multiple linear regression was applied to data collected at three different times during the school year to determine if the two new forms can be used to measure incremental growth across time.

Results suggest that the content of the Basic Skills probes represents a unitary construct, which is basic algebra skills. The two newly created forms of the Basic Skills probe are sensitive to incremental growth in this skill set over time, and are thus valid and reliable for use as progress monitoring measures for students enrolled in Algebra I classes.