Investigating Problem Similarity Through Study of Between-Subject and Within-Subject Experiments

Design problems are used to evaluate students’ abilities, the impact of various teaching approaches and of design methods. Design problems greatly vary in style and subject area in order to accommodate for a wide distribution of disciplines, cultures, and expertise. While design problems are occasionally reused between studies, new design problems are continuously created in order to account for the fact that a design problem cannot be used multiple times on an individual in order to effectively measure one’s abilities to perform design. More specifically, in repeated measures testing, students cannot receive the same design problem multiple times, for this would cause bias; therefore, multiple design problems are needed to allow for repeated measures testing. The nature and structure of these multiple design problems need to be similar or “equivalent” in order to accurately measure students’ abilities to perform in design. In this study, we examine four design problems: peanut shelling, corn husking, coconut harvesting, and a personal alarm clock. We determine whether these problems could be deemed equivalent for the purposes of evaluating student design performance through repeated measures testing. We implemented idea generation sessions using both between-subject and within-subjects approaches. Solutions were evaluated on quantity, quality, novelty, variety, and completeness metrics. The data implies that the Peanut and Corn problems are similar in nature and the Alarm and Coconut problems are also similar in nature; as such, these problem pairings may be used to test differences based on group means.