Using electroencephalography (EEG) to predict design outcomes could be used in many applications as it facilitates the correlation of engagement and cognitive workload with ideation effectiveness. It also establishes a basis for the connection between EEG measurements and common constructs in engineering design research. In this paper, we propose a support vector machine (SVM)-based prediction model for design outcomes using EEG metrics and some demographic factors as predictors. We trained and validated the model with more than 100 concepts, and then evaluated the relationship between EEG data and concept-level measures of novelty, quality, and elaboration. The results characterize the combination of engagement and workload that is correlated with good design outcomes. Findings also suggest that EEG technologies can be used to partially replace or augment traditional ideation metrics and to improve the efficacy of ideation research.
Issue Section:
Design Theory and Methodology
References
1.
Dinar
, M.
, Shah
, J. J.
, Cagan
, J.
, Leifer
, L.
, Linsey
, J.
, Smith
, S. M.
, and Hernandez
, N. V.
, 2015
, “Empirical Studies of Designer Thinking: Past, Present, and Future
,” ASME J. Mech. Des.
, 137
(2
), p. 021101
.2.
Proctor
, R. W.
, and Van Zandt
, T.
, 2008
, Human Factors in Simple and Complex Systems
, 2nd ed., CRC Press
, Boca Raton, FL.3.
Runco
, M. A.
, and Chand
, I.
, 1995
, “Cognition and Creativity
,” Educ. Psychol. Rev.
, 7
(3
), pp. 243
–267
.4.
Carroll
, E. A.
, 2013
, “Quantifying the Personal Creative Experience: Evaluation of Digital Creativity Support Tools Using Self-Report and Physiological Responses
,” Ph.D. thesis
, University of North Carolina at Charlotte, Charlotte, NC
.5.
Amabile
, T. M.
, Conti
, R.
, Coon
, H.
, Lazenby
, J.
, and Herron
, M.
, 1996
, “Assessing the Work Environment for Creativity
,” Acad. Manage. J.
, 39
(5
), pp. 1154
–1184
.6.
Besemer
, S. P.
, 1998
, “Creative Product Analysis Matrix: Testing the Model Structure and a Comparison Among Products: Three Novel Chairs
,” Creativity Res. J.
, 11
(4
), pp. 333
–346
.7.
Gero
, J. S.
, and Mc Neill
, T.
, 1998
, “An Approach to the Analysis of Design Protocols
,” Des. Stud.
, 19
(1
), pp. 21
–61
.8.
Cai
, H.
, Do
, E. Y. L.
, and Zimring
, C. M.
, 2010
, “Extended Linkography and Distance Graph in Design Evaluation: An Empirical Study of the Dual Effects of Inspiration Sources in Creative Design
,” Des. Stud.
, 31
(2
), pp. 146
–168
.9.
Rubio
, S.
, Díaz
, E.
, Martín
, J.
, and Puente
, J. M.
, 2004
, “Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods
,” Appl. Psychol.
, 53
(1
), pp. 61
–86
.10.
Dorta
, T.
, Pérez
, E.
, and Lesage
, A.
, 2008
, “The Ideation Gap: Hybrid Tools, Design Flow and Practice
,” Des. Stud.
, 29
(2
), pp. 121
–141
.11.
Shah
, J. J.
, Vargas-Hernandez
, N.
, and Smith
, S. M.
, 2003
, “Metrics for Measuring Ideation Effectiveness
,” Des. Stud.
, 24
(2
), pp. 111
–134
.12.
Guilford
, J. P.
, 1967
, The Nature of Human Intelligence
, McGraw-Hill
, New York
.13.
Torrance
, E. P.
, 1966
, Torrance Tests of Creative Thinking: Norms-Technical Manual
, Personnel Press
, Princeton, NJ.14.
Dean
, D. L.
, Hender
, J. M.
, Rodgers
, T. L.
, and Santanen
, E.
, 2006
, “Identifying Good Ideas: Constructs and Scales for Idea Evaluation
,” J. Assoc. Inf. Syst.
, 7
(10
), pp. 646
–699
.15.
Nelson
, B. A.
, Wilson
, J. O.
, Rosen
, D.
, and Yen
, J.
, 2009
, “Refined Metrics for Measuring Ideation Effectiveness
,” Des. Stud.
, 30
(6
), pp. 737
–743
.16.
Dippo
, C.
, and Kudrowitz
, B.
, 2015
, “The Effects of Elaboration in Creativity Tests as it Pertains to Overall Scores and How it Might Prevent a Person From Thinking of Creative Ideas During the Early Stages of Brainstorming and Idea Generation
,” ASME
Paper No. DETC2015-46789.17.
O’Quin
, K.
, and Besemer
, S. P.
, 2006
, “Using the Creative Product Semantic Scale as a Metric for Results-Oriented Business
,” Creativity Innovation Manage.
, 15
(1
), pp. 34
–44
.18.
Amabile
, T. M.
, 1982
, “A Consensual Assessment Technique
,” J. Pers. Soc. Psychol.
, 43
(5
), pp. 997
–1013
.19.
Reis
, S. M.
, and Renzulli
, J. S.
, 1991
, “The Assessment of Creative Products in Programs for Gifted and Talented Students
,” Gifted Child Q.
, 35
(3
), pp. 128
–134
.20.
Oman
, S.
, and Tumer
, I. Y.
, 2009
, “The Potential of Creativity Metrics for Mechanical Engineering Concept Design
,” The 17th International Conference on Engineering Design
(ICED 09
), Stanford, CA, Aug. 24–27, Vol. 2
, pp. 145
–156
.21.
Oman
, S. K.
, Tumer
, I. Y.
, Wood
, K.
, and Seepersad
, C.
, 2013
, “A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects
,” Res. Eng. Des.
, 24
(1
), pp. 65
–92
.22.
Srivathsavai
, R.
, Genco
, N.
, Hölttä-Otto
, K.
, and Seepersad
, C. C.
, 2010
, “Study of Existing Metrics Used in Measurement of Ideation Effectiveness
,” ASME
Paper No. DETC2010-28802.23.
Reinig
, B.
, Briggs
, R.
, and Nunamaker
, J.
, 2007
, “On the Measurement of Ideation Quality
,” J. Manage. Inf. Syst.
, 23
(4
), pp. 143
–161
.24.
Fuge
, M.
, Stroud
, J.
, and Agogino
, A.
, 2013
, “Automatically Inferring Metrics for Design Creativity
,” ASME
Paper No. DETC2013-12620.25.
Kudrowitz
, B. M.
, and Wallace
, D.
, 2013
, “Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation
,” J. Eng. Des.
, 24
(2
), pp. 120
–139
.26.
Nisbett
, R. E.
, and Wilson
, T. D.
, 1977
, “Telling More Than We Can Know: Verbal Reports on Mental Processes
,” Psychol. Rev.
, 84
(3
), pp. 231
–259
.27.
Kounios
, J.
, Fleck
, J. I.
, Green
, D. L.
, Payne
, L.
, Stevenson
, J. L.
, Bowden
, E. M.
, and Jung-Beeman
, M.
, 2008
, “The Origins of Insight in Resting-State Brain Activity
,” Neuropsychologia
, 46
(1
), pp. 281
–291
.28.
Fish
, J.
, and Scrivener
, S.
, 1990
, “Amplifying the Mind’s Eye: Sketching and Visual Cognition
,” Leonardo
, 23
(1
), pp. 117
–126
.29.
Goldschmidt
, G.
, 1991
, “The Dialectics of Sketching
,” Creativity Res. J.
, 4
(2
), pp. 123
–143
.30.
Diamond
, A.
, 2013
, “Executive Functions
,” Annu. Rev. Psychol.
, 64
(1
), pp. 135
–168
.31.
Malenka
, R. C.
, Nestler
, E. J.
, and Hyman
, S. E.
, 2009
, “Higher Cognitive Function and Behavioral Control
,” Molecular Neuropharmacology: A Foundation for Clinical Neuroscience
, A.
Sydor
and R. Y.
Brown
, eds., 2nd ed., McGraw-Hill Medical
, New York
, p. 318
.32.
Baddeley
, A. D.
, and Hitch
, G.
, 1974
, “Working Memory
,” Psychol. Learn. Motiv.
, 8
, pp. 47
–89
.33.
Martindale
, C.
, 1999
, “Biological Bases of Creativity
,” Handbook of Creativity
, Cambridge University Press
, Cambridge, UK
, pp. 137
–152
.34.
Simonton
, D. K.
, 2010
, “Creative Thought as Blind-Variation and Selective-Retention: Combinatorial Models of Exceptional Creativity
,” Phys. Life Rev.
, 7
(2
), pp. 156
–179
.35.
Amabile
, T. M.
, 2012
, Componential Theory of Creativity
, Harvard Business School
, Boston, MA
.36.
Alexiou
, K.
, Zamenopoulos
, T.
, Johnson
, J. H.
, and Gilbert
, S. J.
, 2009
, “Exploring the Neurological Basis of Design Cognition Using Brain Imaging: Some Preliminary Results
,” Des. Stud.
, 30
(6
), pp. 623
–647
.37.
Sylcott
, B.
, Cagan
, J.
, and Tabibnia
, G.
, 2013
, “Understanding Consumer Tradeoffs Between Form and Function Through Metaconjoint and Cognitive Neuroscience Analyses
,” ASME J. Mech. Des.
, 135
(10
), p. 101002
.38.
Schaufeli
, W.
, and Bakker
, A. B.
, 2010
, “Defining and Measuring Work Engagement: Bringing Clarity to the Concept
,” Work Engagement: A Handbook of Essential Theory and Research
, US: Psychology Press, New York, pp. 10
–24
.39.
Nguyen
, T. A.
, and Zeng
, Y.
, 2010
, “Analysis of Design Activities Using EEG Signals
,” ASME
Paper No. DETC2010-28477.40.
Nguyen
, T. A.
, and Zeng
, Y.
, 2012
, “Clustering Designers’ Mental Activities Based on EEG Power
,” The Ninth International Symposium on Tools and Methods of Competitive Engineering
(TMCE 2012
), Karlsruhe, Germany, May 7–11.41.
Nguyen
, T. A.
, and Zeng
, Y.
, 2014
, “A Physiological Study of Relationship Between Designer’s Mental Effort and Mental Stress During Conceptual Design
,” Comput. Aided Des.
, 54
, pp. 3
–18
.42.
Carroll
, E. A.
, 2011
, “Convergence of Self-Report and Physiological Responses for Evaluating Creativity Support Tools
,” 8th ACM Conference on Creativity and Cognition
(C&C 11
), Atlanta, Georgia, Nov. 3–6, pp. 455
–456
.43.
Carroll
, E. A.
, and Latulipe
, C.
, 2012
, “Triangulating the Personal Creative Experience: Self-Report, External Judgments, and Physiology
,” Proceedings of Graphics Interface (GI '12
), Toronto, ON, May 28–30, pp. 53
–60
.44.
Toh
, C.
, Miller
, S.
, and Simpson
, T.
, 2015
, “The Impact of Virtual Product Dissection Environments on Student Design Learning and Self-Efficacy
,” J. Eng. Des.
, 26
(1–3
), pp. 48
–73
.45.
Hart
, S. G.
, and Staveland
, L. E.
, 1988
, “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
,” Adv. Psychol.
, 52
, pp. 139
–183
.46.
Hu
, W.-l.
, Booth
, J.
, and Reid
, T.
, 2015
, “Reducing Sketch Inhibition During Concept Generation: Psychophysiological Evidence of the Effect of Interventions
,” ASME
Paper No. DETC2015-47669.47.
Booth
, J. W.
, Taborda
, E. A.
, Ramani
, K.
, and Reid
, T.
, 2016
, “Interventions for Teaching Sketching Skills and Reducing Inhibition for Novice Engineering Designers
,” Des. Stud.
, 43
, pp. 1
–23
.48.
Durand
, F.
, Helms
, M. E.
, Tsenn
, J.
, McAdams
, D. A.
, and Linsey
, J. S.
, 2015
, “In Search of Effective Design Problems for Design Research
,” ASME
Paper No. DETC2015-47701.49.
David Hairston
, W.
, Whitaker
, K. W.
, Ries
, A. J.
, Vettel
, J. M.
, Cortney Bradford
, J.
, Kerick
, S. E.
, and McDowell
, K.
, 2014
, “Usability of Four Commercially-Oriented EEG Systems
,” J. Neural Eng.
, 11
(4
), p. 046018
.50.
Hocevar
, D.
, 1979
, “The Development of the Creative Behavior Inventory (CBI)
,” Annual Meeting of the Rocky Mountain Psychological Association
, Paper No. ED170350
.51.
Berka
, C.
, Levendowski
, D. J.
, Cvetinovic
, M. M.
, Petrovic
, M. M.
, Davis
, G.
, Lumicao
, M. N.
, Zivkovic
, V. T.
, Popovic
, M. V.
, and Olmstead
, R.
, 2004
, “Real-Time Analysis of EEG Indexes of Alertness, Cognition, and Memory Acquired With a Wireless EEG Headset
,” Int. J. Hum. Comput. Interact.
, 17
(2
), pp. 151
–170
.52.
Berka
, C.
, Levendowski
, D. J.
, Lumicao
, M. N.
, Yau
, A.
, Davis
, G.
, Zivkovic
, V. T.
, Olmstead
, R. E.
, Tremoulet
, P. D.
, and Craven
, P. L.
, 2007
, “EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks
,” Aviat., Space, Environ. Med.
, 78
(5
), pp. B231
–B244
.53.
de Guinea
, A. O.
, Titah
, R.
, and Léger
, P.-M.
, 2013
, “Measure for Measure: A Two Study Multi-Trait Multi-Method Investigation of Construct Validity in IS Research
,” Comput. Hum. Behav.
, 29
(3
), pp. 833
–844
.54.
Johnson
, R. R.
, Popovic
, D. P.
, Olmstead
, R. E.
, Stikic
, M.
, Levendowski
, D. J.
, and Berka
, C.
, 2011
, “Drowsiness/Alertness Algorithm Development and Validation Using Synchronized EEG and Cognitive Performance to Individualize a Generalized Model
,” Biol. Psychol.
, 87
(2
), pp. 241
–250
.55.
Ries
, A. J.
, Touryan
, J.
, Vettel
, J.
, McDowell
, K.
, and Hairston
, W. D.
, 2014
, “A Comparison of Electroencephalography Signals Acquired From Conventional and Mobile Systems
,” J. Neurosci. Neuroeng.
, 3
(1
), pp. 10
–20
.56.
Oberauer
, K.
, Süß
, H. M.
, Schulze
, R.
, Wilhelm
, O.
, and Wittmann
, W. W.
, 2000
, “Working Memory Capacity—Facets of a Cognitive Ability Construct
,” Pers. Individ. Differ.
, 29
(6
), pp. 1017
–1045
.57.
Sun
, X.
, Zhang
, X.
, Chen
, X.
, Zhang
, P.
, Bao
, M.
, Zhang
, D.
, Chen
, J.
, He
, S.
, and Hu
, X.
, 2005
, “Age-Dependent Brain Activation During Forward and Backward Digit Recall Revealed by fMRI
,” Neuroimage
, 26
(1
), pp. 36
–47
.58.
Stemler
, S. E.
, 2004
, “A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability
,” Pract. Assess., Res. Eval.
, 9
(4
).59.
Neuendorf
, K. A.
, 2002
, The Content Analysis Guidebook
, SAGE
, Thousand Oaks, CA.60.
Kroeker
, M.
, Lacy
, M.
, Mark
, A.
, and Van De Loo
, J.
, 1996
, “Interrater Reliability in the Assessment of Sitting Balance in Hemiparetic Patients
,” Masters thesis
, University of Osteopathic Medicine and Health Sciences
, Des Moines, IA.61.
LeBreton
, J. M.
, and Senter
, J. L.
, 2008
, “Answers to 20 Questions About Interrater Reliability and Interrater Agreement
,” Organ. Res. Methods
, 11
(4
), pp. 815
–852
.62.
Kershaw
, T. C.
, Peterson
, R. L.
, McCarthy
, M. A.
, Young
, A. P.
, Seepersad
, C. C.
, Williams
, P. T.
, Hölttä-Otto
, K.
, and Bhowmick
, S.
, 2015
, “A Cross-Sectional and Longitudinal Examination of the Development of Innovation Capability in Undergraduate Engineering Students
,” ASME
Paper No. DETC2015-47650.63.
Charyton
, C.
, Jagacinski
, R. J.
, and Merrill
, J. A.
, 2008
, “CEDA: A Research Instrument for Creative Engineering Design Assessment
,” Psychol. Aesthetics, Creativity, Arts
, 2
(3
), pp. 147
–154
.64.
Maletic
, J. I.
, and Marcus
, A.
, 2000
, “Data Cleansing: Beyond Integrity Analysis
,” 2000 Conference on Information Quality (IQ-2000)
, pp. 200
–209
.65.
Xiong
, H.
, and Pandey
, G.
, 2006
, “Enhancing Data Analysis With Noise Removal
,” IEEE Trans. Knowl. Data Eng.
, 18
(3
), pp. 304
–319
.66.
Lotte
, F.
, Congedo
, M.
, Lécuyer
, A.
, Lamarche
, F.
, and Arnaldi
, B.
, 2007
, “A Review of Classification Algorithms for EEG-Based Brain-Computer Interfaces
,” J. Neural Eng.
, 4
(2
), pp. R1
–R13
.67.
Sohaib
, A. T.
, Qureshi
, S.
, Hagelbäck
, J.
, Hilborn
, O.
, and Jerčić
, P.
, 2013
, “Evaluating Classifiers for Emotion Recognition Using EEG
,” Foundations of Augmented Cognition: 7th International Conference, Part of HCI International 2013
(AC 2013
), Las Vegas, NV, July 21–26, 2013
, D. D.
Schmorrow
and C. M.
Fidopiastis
, eds., Springer, Berlin, pp. 492
–501
.68.
Singla
, R.
, Chambayil
, B.
, Khosla
, A.
, and Santosh
, J.
, 2011
, “Comparison of SVM and ANN for Classification of eye Events in EEG
,” J. Biomed. Sci. Eng.
, 4
(01
), pp. 62
–69
.69.
Beal
, C. R.
, and Cirett Galán
, F.
, 2012
, “EEG Estimates of Cognitive Workload and Engagement Predict Math Problem Solving Outcomes
,” International Conference on User Modeling, Adaptation, and Personalization, Society for Research on Educational Effectiveness
(UMAP 2012
), Montreal, QC, July 16–20, pp. 51
–62
.70.
Ghali
, R.
, and Frasson
, C.
, 2015
, “Classification and Regression of Learner’s Scores in Logic Environment
,” J. Educ. Training Stud.
, 3
(5
), pp. 242
–253
.71.
Myrden
, A.
, and Chau
, T.
, 2015
, “Effects of User Mental State on EEG-BCI Performance
,” Front. Hum. Neurosci.
, 9
, p. 308
.72.
Borghini
, G.
, Astolfi
, L.
, Vecchiato
, G.
, Mattia
, D.
, and Babiloni
, F.
, 2014
, “Measuring Neurophysiological Signals in Aircraft Pilots and Car Drivers for the Assessment of Mental Workload, Fatigue and Drowsiness
,” Neurosci. Biobehav. Rev.
, 44
, pp. 58
–75
.73.
Kohavi
, R.
, 1995
, “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
,” International Joint Conference on Artificial Intelligence
(IJCAI'95
), Montreal, QC, Aug. 20–25, Vol. 14
, pp. 1137
–1143
.74.
Wu
, C. H.
, Cheng
, Y.
, Ip
, H. M.
, and Mcbride-Chang
, C.
, 2005
, “Age Differences in Creativity: Task Structure and Knowledge Base
,” Creativity Res. J.
, 17
(4
), pp. 321
–326
.75.
Agogué
, M.
, Poirel
, N.
, Pineau
, A.
, Houdé
, O.
, and Cassotti
, M.
, 2014
, “The Impact of Age and Training on Creativity: A Design-Theory Approach to Study Fixation Effects
,” Thinking Skills Creativity
, 11
, pp. 33
–41
.76.
Purcell
, A. T.
, and Gero
, J. S.
, 1996
, “Design and Other Types of Fixation
,” Des. Stud.
, 17
(4
), pp. 363
–383
.77.
Sobek
, D. K.
, and Jain
, V. K.
, 2007
, “Relating Design Process to Quality: A Virtual Design of Experiments Approach
,” ASME J. Mech. Des.
, 129
(5
), pp. 483
–490
.78.
Chusilp
, P.
, and Jin
, Y.
, 2004
, “Cognitive Modeling of Iteration in Conceptual Design
,” ASME
Paper No. DETC2004-57521.79.
Chusilp
, P.
, and Jin
, Y.
, 2006
, “Impact of Mental Iteration on Concept Generation
,” ASME J. Mech. Des.
, 128
(1
), pp. 14
–25
.80.
Liu
, Z.
, Yang
, D.-S.
, Wen
, D.
, Zhang
, W.-M.
, and Mao
, W.
, 2011
, “Cyber-Physical-Social Systems for Command and Control
,” IEEE Intell. Syst.
, 26
(4
), pp. 92
–96
.81.
Borgman
, C. L.
, Abelson
, H.
, Dirks
, L.
, Johnson
, R.
, Koedinger
, K. R.
, Linn
, M. C.
, Lynch
, C. A.
, Oblinger
, D. G.
, Pea
, R. D.
, Salen
, K.
, Smith
M. S.
, and Szalay
, A.
, 2008
, “Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge. A 21st Century Agenda for the National Science Foundation
,” Report of the NSF Task Force on Cyberlearning, Report No. nsf08204
.82.
Linsey
, J. S.
, Tseng
, I.
, Fu
, K.
, Cagan
, J.
, Wood
, K. L.
, and Schunn
, C.
, 2010
, “A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty
,” ASME J. Mech. Des.
, 132
(4
), p. 041003
.83.
Viswanathan
, V.
, Atilola
, O.
, Esposito
, N.
, and Linsey
, J.
, 2014
, “A Study on the Role of Physical Models in the Mitigation of Design Fixation
,” J. Eng. Des.
, 25
(1–3
), pp. 25
–43
.Copyright © 2017 by ASME
You do not currently have access to this content.
