Usability of in-vehicle systems has become increasingly important for ease of operations and safety of driving. The user interface (UI) of in-vehicle systems is a critical focus of usability study. This paper studies how to use advanced computational, physiology- and behavior-based tools and methodologies to determine affective/emotional states and behavior of an individual in real time and in turn how to adapt the human-vehicle interaction to meet users' cognitive needs based on the real-time assessment. Specifically, we set up a set of physiological sensors that are capable of collecting EEG, facial EMG, skin conductance response, and respiration data and a set of motion sensing and tracking equipment that is capable of capturing eye ball movement and objects which the user is interacting with. All hardware components and software are integrated into an augmented sensor platform that can perform as “one coherent system” to enable multimodal data processing and information inference for context-aware analysis of emotional states and cognitive behavior based on the rough set inference engine. Meanwhile subjective data are also recorded for comparison. A usability study of in-vehicle system UI is shown to demonstrate the potential of the proposed methodology.
Issue Section:
Research Papers
References
1.
Kim
, S.
, Dey
, A. K.
, Lee
, J.
, and Forlizzi
, J.
, 2011
, “Usability of Car Dashboard Displays for Elder Drivers
,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
, ACM, Vancouver, BC, Canada, pp. 493
–502
.2.
Harvey
, C.
, Stanton
, N. A.
, Pickering
, C. A.
, McDonald
, M.
, and Zheng
, P.
, 2011
, “A Usability Evaluation Toolkit for In-Vehicle Information Systems (IVISs)
,” Appl. Ergon.
, 42
(4
), pp. 563
–574
.10.1016/j.apergo.2010.09.0133.
Pauzié
, A.
, 2008
, “Evaluating Driver Mental Workload Using the Driving Activity Load Index (DALI)
,” European Conference on Human Centred Design for Intelligent Transport Systems
, Lyon, France.4.
Young
, K.
, and Regan
, M.
, 2007
, “Driver distraction: A review of the literature
,” Distracted Driving
, I. J.
Faulks
, M.
Regan
, M.
Stevenson
, J.
Brown
, A.
Porter
, and J. D.
Irwin
, eds., Australasian College of Road Safety
, Sydney, NSW
, pp. 379
–405
.5.
Hix
, D.
, Swan
, E. J.
, Gabbard
, J. L.
, McGee
, M.
, Durbin
, J.
, and King
, T.
, “User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment
,” Proc. IEEE VR'99 Conference, IEEE Computer Society Press
, pp. 96
–103
.6.
Gabbard
, J. L.
, Swan
, J. E.
, Hix
, D.
, Lanzagorta
, M.
, Livingston
, M.
, Brown
, D.
, and Julier
, S.
, 2002
, “Usability Engineering: Domain Analysis Activities for Augmented Reality Systems
,” 9th Annual Engineering Reality of Virtual Reality Conference
, San Jose, CA.7.
Rosson
, M. B.
, and Carroll
, J. M.
, 2002
, Usability Engineering: Scenario-Based Development of Human-Computer Interaction
, Academic Press
, San Fancisco
.8.
Hix
, D.
, and Hartson
, H. R.
, 1993
, Developing User Interfaces: Ensuring Usability through Product & Process
, John Wiley and Sons, Inc.
, New York
.9.
Gabbard
, J. L.
, Hix
, D.
, Swan
, J. E.
, Livingston
, M. A.
, Höllerer
, T. H.
, Julier
, S. J.
, Brown
, D.
, and Baillot
, Y.
, 2003
, “Usability Engineering for Complex Interactive Systems Development
,” Human Systems Integration Symposium
, Vienna, VA.10.
Zhou
, F.
, Xu
, Q.
, and Jiao
, R.
, 2011
, “Fundamentals of Product Ecosystem Design for User Experience
,” Res. Eng. Des.
, 22
(1
), pp. 43
–61
.10.1007/s00163-010-0096-z11.
Zhou
, F.
, Ji
, Y. J.
, and Jiao
, R. J.
, 2012
, “Affective and Cognitive Design for Mass Personalization: Status and Prospect
,” J. Intell. Manuf.
, 24
(5
), pp. 1047
–1069
10.1007/s10845-012-0673-2.12.
Grandjean
, D.
, Sander
, D.
, and Scherer
, K. R.
, 2008
, “Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization
,” Conscious Cogn.
, 17
(2
), pp. 484
–495
.10.1016/j.concog.2008.03.01913.
Stone
, A. A.
, and Shiffman
, S.
, 1994
, “Ecological Momentary Assessment (EMA) in Behavioral Medicine
,” Ann. Behav. Med.
, 16
(3
), pp. 199
–202
.14.
Paas
, F.
, Tuovinen
, J. E.
, Tabbers
, H. K.
, and Van Gerven
, P. W. M.
, 2003
, “Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
,” Educ. Psychol.
, 38
(1
), pp. 63
–71
.10.1207/S15326985EP3801_815.
Haapalainen
, E.
, Kim
, S.
, Forlizzi
, J. F.
, and Dey
, A. K.
, 2010
, “Psycho-Physiological Measures for Assessing Cognitive Load
,” Proceedings of the 12th ACM international conference on Ubiquitous computing
, ACM, Copenhagen, Denmark, pp. 301
–310
.16.
Ericsson
, K.
, and Simon
, H.
, 1993
, Protocol Analysis: Verbal Reports as Data
, MIT Press
, Boston, MA
.17.
Detenber
, B. H.
, 2001
, “Measuring Emotional Responses in Human Factors Research: Some Theoretical and Practical Considerations
,” Proceedings of the International Conference on Affective Human Factors Design
, M.
Helander
, H.
Khalid
, and M.
Tham
, eds., Asean Academic Press
, London
, pp. 124
–130
.18.
Annett
, J.
, 2002
, “Subjective Rating Scales: Science or Art?
,” Ergonomics
, 45
(14
), pp. 966
–987
.10.1080/0014013021016695119.
Reid
, G. B.
, Shingledecker
, C. A.
, Nygren
, T. E.
, and Eggemeier
, F. T.
, 1981
, “Development of Multidimensional Subjective Measures of Workload
,” IEEE International Conference on Cybernetics & Society
, Atlanta, GA.20.
Hart
, S. G.
, and Staveland
, L. E.
, 1988
, “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
,” Human Mental Workload
, P. A.
Hancock
, and N.
Meshkati
, eds., North-Holland Press
, Amsterdam
.21.
Nielsen
, J.
, 1994
, Usability Engineering
, Morgan Kaufmann
, San Francisco, CA
.22.
Mital
, A.
, and Govindaraju
, M.
, 1999
, “Is It Possible to Have a Single Measure for All Work?
,” Int. J. Indus. Eng. Theory
, 6
(3
), pp. 190
–195
.23.
Young
, K. L.
, Regan
, M. A.
, and Lee
, J. D.
, 2009
, “Measuring the Effects of Driver Distraction: Direct Driving Performance Methods and Measures
,” Driver Distraction: Theory, Effects and Mitigation
, M. A.
Regan
, J. D.
Lee
: K. L.
Young
, eds., CRC Press
, Boca Raton, FL
, pp. 85
–105
.24.
Wittmann
, M.
, Kiss
, M.
, Gugg
, P.
, Steffen
, A.
, Fink
, M.
, Pöppel
, E.
, and Kaniya
, H.
, 2006
, “Effect of Display Position of a Visual In-Vehicle Task on Simulated Driving
,” Appl. Ergon.
, 37
(2
), pp. 187
–199
.10.1016/j.apergo.2005.06.00225.
Chiang
, D. P.
, Brooks
, A. M.
, and Weir
, D. H.
, 2004
, “On the Highway Measures of Driver Glance Behavior With An Example Automobile Navigation System
,” Appl. Ergon.
, 35
(3
), pp. 215
–223
.10.1016/j.apergo.2004.01.00526.
Liu
, C.-C.
, Doong
, J.-L.
, Hsu
, C.-C.
, Huang
, W.-S.
, and Jeng
, M.-C.
, 2009
, “Evidence for the Selective Attention Mechanism and Dual-Task Interference
,” Appl. Ergon.
, 40
(3
), pp. 341
–347
.10.1016/j.apergo.2008.11.01427.
Zhou
, F.
, Qu
, X.
, Helander
, M. G.
, and Jiao
, J.
, 2011
, “Affect Prediction From Physiological Measures Via Visual Stimuli
,” Int. J. Hum.-Comput. Stud.
, 69
(12
), pp. 801
–819
.10.1016/j.ijhcs.2011.07.00528.
Schiano
, D. J.
, Ehrlich
, S. M.
, and Sheridan
, K.
, 2004
, “Categorical Imperative NOT: Facial Affect is Perceived Continuously
,” SIGCHI Conference on Human Factors in Computing Systems
, Vienna, Austria.29.
Bailenson
, J. N.
, Pontikakis
, E. D.
, Mauss
, I. B.
, Gross
, J. J.
, Jabon
, M. E.
, Hutcherson
, C. A. C.
, Nass
, C.
, and Oliver
, J.
, 2008
, “Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses
,” Int. J. Hum.-Comput. Stud.
, 66
(5
), pp. 303
–317
.10.1016/j.ijhcs.2007.10.01130.
Fairclough
, S. H.
, 2009
, “Fundamentals of Physiological Computing
,” Interact. Comput.
, 21
(1-2
), pp. 133
–145
.10.1016/j.intcom.2008.10.01131.
Mandryk
, R.
, and Atkins
, M.
, 2007
, “A Fuzzy Physiological Approach for Continuously Modeling Emotion During Interaction With Play Technologies
,” Int. J. Hum.-Comput. Stud.
, 65
(4
), pp. 329
–347
.10.1016/j.ijhcs.2006.11.01132.
Niedermeyer
, E.
, and Lopes Da Silva
, F.
, 2004
, Electroencephalography: Basic Principles, Clinical Applications, and Related Fields
, Lippincott Williams & Wilkins
, Philadelphia, PA
.33.
Bos
, D. O.
, 2008
, “EEG-Based Emotion Recognition: The Influence of Visual and Auditory Stimuli
.” Available from: http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf34.
Lang
, P. J.
, Greenwald
, M. K.
, Bradley
, M. M.
, and Hamm
, A. O.
, 1993
, “Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions
,” Psychophysiology
, 30
(3
), pp. 261
–273
.10.1111/j.1469-8986.1993.tb03352.x35.
Ablassmeier
, M.
, Poitschke
, T.
, Wallhoff
, F.
, Bengler
, K.
, and Rigoll
, G.
, 2007
, “Eye Gaze Studies Comparing Head-Up and Head-Down Displays in Vehicles
,” IEEE International Conference on Multimedia and Expo
, Beijing, China.36.
Pawlak
, Z.
, 1991
, Rough Sets: Theoretical Aspects of Reasoning About Data
, Kluwer
, Dordrecht, The Netherlands
.37.
Duntsch
, I.
, and Gediga
, G.
, 1998
, “Uncertainty Measures of Rough Set Prediction
,” AI Commun.
, 106
(1
), pp. 109
–137
.38.
Picard
, R. W.
, Vyzas
, E.
, and Healey
, J.
, 2001
, “Toward Machine Emotional Intelligence: Analysis of Affective Physiological State
,” IEEE Trans. Pattern Anal. Mach. Intell.
, 23
(10
), pp. 1175
–1191
.10.1109/34.95460739.
Welch
, P. D.
, 1967
, “The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms
,” IEEE Trans. Audio Electroacoust.
, AU-15
, pp. 70
–73
.10.1109/TAU.1967.116190140.
Fodor
, I. K.
, 2002
, “A Survey of Dimension Reduction Techniques
,” Lawrence Livermore National Laboratory, Livermore, CA, Technical Report No. UCRL-ID-148494.41.
Martinez
, A. M.
, and Kak
, A. C.
, 2001
, “PCA Versus LDA
,” IEEE Trans. Pattern Anal. Mach. Intell.
, 23
(2
), pp. 228
–233
.10.1109/34.90897442.
Fisher
, R. A.
, 1938
, “The Statistical Utilization of Multiple Measurements
,” Ann. Eugen.
, 8
(4
), pp. 376
–386
.10.1111/j.1469-1809.1938.tb02189.x43.
Bazan
, J.
, and Szczuka
, M.
, 2000
, RSES and RSESlib—A Collection of Tools for Rough Set Computations
, Lecture Notes in Artificial Intelligence, Springer-Verlag
, Berlin, Heidelberg
, pp. 592
–601
.44.
Gora
, G.
, and Wojna
, A.
, 2002
, “Riona: A New Classification System Combining Rule Induction and Instance-Based Learning
,” Fund. Inform.
, 51
(4
), pp. 369
–390
.45.
Cacioppo
, J. T.
, and Tassinary
, L. G.
, 1990
, “Inferring Psychological Significance From Physiological Signals
,” Am. Psychol.
, 45
(1
), pp. 16
–28
.10.1037/0003-066X.45.1.1646.
Gu
, Y.
, Tan
, S. L.
, Wong
, K. J.
, Ho
, M.-H. R.
, and Qu
, L.
, 2010
, “A GMM Based 2-Stage Architecture for Multi-Subject Emotion Recognition Using Physiological Responses
,” The 1st Augmented Human International Conference
, Megève, France.47.
Zhou
, F.
, and Jiao
, R. J.
, 2013
, “A Hierarchical Bayesian Model for Affective Parameter Shaping in User Experience Prospect Evaluation
,” IEEE Trans. Affect. Comput.
(submitted).48.
Victor
, T. W.
, Harbluk
, J. L.
, and Engstro
, J. A.
, 2005
, “Sensitivity of Eye-Movement Measures to In-Vehicle Task Difficulty
,” Transp. Res. Part F
, 8
(2
), pp. 167
–190
.10.1016/j.trf.2005.04.01449.
Dingus
, T. A.
, Klauer
, S. G.
, Neale
, V. L.
, Petersen
, A.
, Lee
, S. E.
, and Sudweeks
, J.
, 2006
, “The 100-Car Naturalistic Driving Study, Phase II—Results of the 100-Car Field Experiment
,” National Highway Traffic Safety Administration, Washington, DC, Paper No. DOT HS 810-593.50.
Lee
, J. D.
, 2009
, “Can Technology Get Your Eyes Back on the Road?
,” Science
, 324
, pp. 344
–346
.10.1126/science.116808551.
Roman
, B.
, Pavel
, S.
, Miroslav
, P.
, Petr
, V.
, and Lubomír
, P.
, 2001
, “Fatigue Indicators of Drowsy Drivers Based on Analysis of Physiological Signals
,” Medical Data Analysis
, J.
Crespo
, V.
Maojo
, and F.
Martin
, eds., Springer
, Berlin/Heidelberg
, pp. 62
–68
.52.
Grace
, R.
, Byrne
, V. E.
, Bierman
, D. M.
, Legrand
, J.-M.
, Gricourt
, D.
, Davis
, B. K.
, Staszewski
, J. J.
, and Carnahan
., B.
, 1998
, “A Drowsy Driver Detection System for Heavy Vehicles
,” 17th Digital Avionics Systems Conference
, Seattle, WA.Copyright © 2014 by ASME
You do not currently have access to this content.
