The Brave New World of “Innovative” Items @ CREATE Lab, 196 Mercer St.

06/15/2012 10:00 am – 06/15/2012 12:00 pm
In Sciences of the Artificial, Herbert Simon wrote: “Designing assessment systems, like designing buildings and software, is an exercise in design under constraints.” High stakes assessments have become more common and widespread and the amalgamation of stakeholder demands require that high stakes assessments use new technology, follow RTI, test more and younger children than ever before, have universal design, be conceptually richer, and produce a wide array of diagnostic information. All of these factors greatly complicate the design process. Demands have pushed the design of items to be much more complex, and time consuming compared to traditional design of highstakes assessment items. While the next generation of assessments is currently being developed, much of the basic science still needs to be done to make these assessments sound and meet stakeholder demands. We provide an overview of conception  innovative items. In particular we give a mathematical framework of the binary relations on sets that can be used to score items. The binary relation framework decomposes items into sets and provides a mathematical basis for assigning full and partial credit. We explore possible difficulties both typical and non-typical learners will experience when completing these tasks. We call for integrative research into novel item and test construction, including competent and cooperative insights from interface designers and developmental experts, above and beyond the use of the traditional collaboration of content experts and psychometricians. Some real items, including example innovative items, are used to illustrate. Bios: Jay Verkuilen is Assistant Professor of Educational Psychology at the City University of New York Graduate Center. He obtained his PhD in quantitative psychology in 2007 from the University of Illinois. He has done theoretical and applied research in psychometrics, quantitative methodology and mathematical psychology published in journals such as the Journal of Mathematical Psychology, Psychological Methods and the Journal of Educational and Behavioral Statistics. His applied work has involved collaborations on topics such as the psycholinguistics of aphasia, tai ji and health, political psychology, experimental game theory and high stakes assessment. Amy Racanello is a post-doctoral fellow in Neuropsychiatry at Columbia University Medical Center. She obtained her PhD in school psychology in 2012 from the City University of New York Graduate Center’s PhD Program in Educational Psychology. Prior to her work at the Graduate Center she earned her masters and advanced certificate in school psychology from Brooklyn College. Amy has extensive experience as a tutor for high stakes assessments. In addition to her assessment work, she has been published and presented on topics including otitis media, therapeutic yoga, and managed care’s relationship to mental health.