A multimodal teleoperation interface is introduced, featuring an integrated virtual reality (VR) based simulation augmented by sensors and image processing capabilities onboard the remotely operated vehicle. The proposed virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multimodal control interface. VR addresses the typical limitations of video based teleoperation caused by signal lag and limited field of view, allowing the operator to navigate in a continuous fashion. The vehicle incorporates an onboard computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and a real-state tracking system enable temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle’s teleoperated state. The system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The VR based multimodal teleoperation interface is expected to be more adaptable and intuitive when compared with other interfaces.
Issue Section:
Technical Briefs
1.
Knutzon
, J. S.
, Sannier
, A. V.
, and Oliver
, J. H.
, 2004, “An Immersive Approach to Command and Control
,” Journal of Battlefield Technology
, 7
(1
), pp. 37
–42
.2.
Lin
, Q.
, and Kuo
, C.
, 1999, “Assisting the Teleoperation of an Unmanned Underwater Vehicle Using a Synthetic Subsea Scenario
,” Presence: Teleoperators and Virtual Environments
, 8
(5
), pp. 520
–530
.3.
Knutzon
, J. S.
, 2003, “Tracking and Control Design for a Virtual Reality Aided Teleoperation System
,” MS thesis, Iowa State University, Ames, IA.4.
Ruff
, H. A.
, Narayanan
, S.
, and Draper
, M. H.
, 2002, “Human Interaction With Levels of Automation and Decision-Aid Fidelity in the Supervisory Control of Multiple Simulated Unmanned Air Vehicles
,” Presence: Teleoperators and Virtual Environments
, 11
(4
), pp. 335
–351
.5.
Dunkler
, O.
, Mitchell
, C. M.
, Govindaraj
, T.
, and Ammons
, J. C.
, 1988, “The Effectiveness of Supervisory Control Strategies in Scheduling Flexible Manufacturing Systems
,” IEEE Trans. Syst. Man Cybern.
0018-9472, 18
(2
), pp. 223
–237
.6.
Fong
, T.
, and Thorpe
, C.
, 2001, “Vehicle Teleoperation Interfaces
,” Auton. Rob.
, 11
, pp. 9
–18
. 0929-55937.
Hainsworth
, D. W.
, 2001, “Teleoperation User Interfaces for Mining Robotics
,” Auton. Rob.
, 11
, pp. 19
–28
. 0929-55938.
Kim
, W. S.
, Hannaford
, B.
, and Bejczy
, A. K.
, 1992, “Force-Reflection and Shared Compliant Control in Operating Telemanipulators With Time Delay
,” IEEE Trans. Rob. Autom.
1042-296X, 8
(2
), pp. 176
–185
.9.
Grant
, R.
, 2002, “Reach Forward: What Happened to the Idea That Execution Authority Should be Delegated to the Lowest Possible Level
?,” Air Force Magazine, Air Force Association
, 85
(10
), pp. 42
–47
.10.
Glumm
, M.
, Kilduff
, P.
, and Masley
, A.
, 1992, “A Study on the Effects of Lens Focal Length on Remote Driver Performance
,” Army Research Laboratory, Technical Report No. ARL-TR-25.11.
Walter
, B. E.
, 2003, “Virtual Reality Aided Teleoperation
,” MS thesis, Iowa State University, Ames, IA.12.
Jarvis
, R. A.
, 1999, “Sensor Rich Teleoperation of an Excavating Machine
,” Proceedings of Conference on Field and Service Robotics
, pp. 238
–243
.13.
Ricks
, B.
, Nielsen
, C. W.
, and Goodrich
, M. A.
, 2004, “Ecological Displays for Robot Interaction: A New Perspective
,” Proceedings of Conference on Intelligent Robots and Systems
, Sendai, Japan, Vol. 3
, pp. 2855
–2860
.14.
Sugimoto
, M.
, Kagotani
, G.
, Nii
, H.
, Shiroma
, N.
, Inami
, M.
, and Matsuno
, F.
, 2005, “Time Follower’s Vision: A Teleoperation Interface With Past Images
,” IEEE Comput. Graphics Appl.
, 25
(1
), pp. 54
–63
. 0272-171615.
Hughes
, S. B.
, and Lewis
, M.
, 2005, “Task-Driven Camera Operations for Robotic Exploration
,” IEEE Trans. Syst. Man Cybern., Part A. Syst. Humans
1083-4427, 35
(4
), pp. 513
–522
.16.
Milgram
, P.
, Rastogi
, A.
, and Grodski
, J. J.
, 1995, “Telerobotic Control Using Augmented Reality
,” Fourth IEEE International Workshop on Robot and Human Communication
, pp. 21
–29
.17.
Brujic-Okretic
, V.
, Guillemaut
, J. Y.
, Hitchin
, L. J.
, Michielen
, M.
, and Parker
, G. A.
, 2003, “Remote Vehicle Maneuvering Using Augmented Reality
,” International Conference on Visual Information Engineering Ideas, Applications, Experience
, pp. 186
–189
.18.
Lawson
, S. W.
, Pretlove
, J. R. G.
, Wheeler
, A. C.
, and Parker
, G. A.
, 2002, “Augmented Reality as a Tool to Aid the Telerobotic Exploration and Characterization of Remote Environments
,” Presence: Teleoperators and Virtual Environments
, 11
(4
), pp. 352
–367
.19.
Fuchs
P.
, Nashaashibi
F.
, and Maman
D.
, 2002, “Assistance for Telepresence by Stereovision Based Augmented Reality and Interactivity in 3D Space
,” Presence: Teleoperators and Virtual Environments
, 11
(5
), pp. 525
–535
.20.
Calhoun
, G. L.
, Draper
, M. H.
, Abernathy
, M. F.
, Delgado
, F.
, and Patzek
, M.
, 2005, “Synthetic Vision System for Improving Unmanned Aerial Vehicle Operator Situation Awareness
,” Proceedings of SPIE Enhanced and Synthetic Vision
, Vol. 5802
, pp. 219
–230
.21.
Fong
, T. W.
, Thorpe
, C.
, and Baur
, C.
, 2001, “Advanced Interfaces for Vehicle Teleoperation: Collaborative Control, Sensor Fusion Displays, and Remote Driving Tools
,” Auton. Rob.
, 11
, pp. 77
–85
. 0929-559322.
Fong
, T.
, Conti
, F.
, Grange
, S.
, and Baur
, C.
, 2000, “Novel Interfaces for Remote Driving: Gesture, Haptic and PDA
,” Proceedings of SPIE Telemanipulator and Telepresence Technologies VII
.23.
Milgram
, P.
, and Ballantyne
, J.
, 1997, “Real World Teleoperation via Virtual Environment Modeling
,” International Conference on Artificial Reality and Tele-existence
.24.
Monferrer
, A.
, and Bonyuet
, D.
, 2002, “Cooperative Robot Teleoperation Through Virtual Reality Interfaces
,” Proceedings of the Sixth International Conference on Information Visualization
, pp. 243
–248
.25.
Hine
, B. P.
, Stoker
, C.
, Sims
, M.
, Rasmussen
, D.
, Hontalas
, P.
, Fong
, T.
, Steele
, J.
, Barch
, D.
, Andersen
, D.
, Miles
, E.
, and Nygren
, E.
, 1994, “The Application of Telepresence and Virtual Reality to Subsea Exploration
,” Second Workshop on Mobile Robots for Subsea Environments
.26.
Thalmann
, D.
, Salamin
, P.
, Ott
, R.
, Gutiérrez
, M.
, and Vexo
, F.
, 2006, “Computer and Information Sciences: Advanced Mixed Reality Technologies for Surveillance and Risk Prevention Applications
,” International Symposium on Computer and Information Sciences (ISCIS)
, pp. 13
–23
.27.
Sheik-Nainar
, M. A.
, Kaber
, D. B.
, and Chow
, M.
, 2005, “Control Gain Adaptation in Virtual Reality Mediated Human-Telerobot Interaction
,” Int. J. Hum. Factors Manufact.
1090-8471, 15
, pp. 259
–274
.28.
Livatino
, S.
, and Privitera
, F.
, 2006, “3D Visualization Technologies for Teleguided Robots
,” Proceedings of the ACM Symposium on Virtual Reality Software and Technology
, New York, pp. 240
–243
.29.
Demiralp
, C.
, Jackson
, C. D.
, Karelitz
, D. B.
, Zhang
, S.
, and Laidlaw
, D. H.
, 2006, “CAVE and Fishtank Virtual-Reality Displays: A Qualitative and Quantitative Comparison
,” IEEE Trans. Vis. Comput. Graph.
1077-2626, 12
(3
), pp. 323
–330
.30.
Walter
, B. E.
, Knutzon
, J. S.
, Sannier
, A. S.
, and Oliver
, J. H.
, 2004, “VR Aided Control of UAVs
,” Third AIAA Unmanned Unlimited Technical Conference, Workshop and Exhibit
, Paper No. 6320.31.
Collins
, R. T.
, Cheng
, Y.
, Jaynes
, C.
, Stolle
, F.
, Wang
, X.
, Hanson
, A. R.
, and Riseman
, E. M.
, 1995, “Site Model Acquisition and Extension from Aerial Images
,” Fifth International Conference on Computer Vision
, pp. 888
–893
.32.
Nguyen
, A.
, Bualat
, M.
, Edwards
, J.
, Flueckiger
, L.
, Neveu
, C.
, Schwehr
, K.
, Wagner
, M. D.
, and Zbinden
, E.
, 2001, “Virtual Reality Interfaces for Visualization and Control of Remote Vehicles
,” Autonomous Robots
, Kluwer Academic
, Dordrecht
, Vol. 11
, pp. 59
–68
.33.
Kadavasal
, M. S.
, and Oliver
, J. H.
, 2007, “Sensor Enhanced Virtual Reality Teleoperation in Dynamic Environment
,” Proceedings of the IEEE Virtual Reality Conference
, pp. 297
–298
.34.
Meier
, R.
, Fong
, T. W.
, Thorpe
, C.
, and Baur
, C.
, 1999, “Sensor Fusion Based User Interface for Vehicle Teleoperation
,” International Conference on Field and Service Robotics
.35.
Kroll
, C. V.
, and Roland
, R. D.
, Jr., 1970, “A Preview-Predictor Model of Driver Behavior in Emergency Situations
,” Cornell Aeronautical Laboratory, Inc., Bureau of Public Roads, Traffic Systems Research Division, Report No. CAL VJ-2251-V-6.36.
Wormell
, D.
, and Foxlin
, E.
, 2003, “Advancements in 3D Interactive Devices for Virtual Environments
,” Joint International Immersive Projection Technologies /Eurographics Workshop on Virtual Environments
.37.
Balakrishnan
, G.
, Sainarayanan
, G.
, Nagarajan
, R.
, and Yaacob
, S.
, 2004, “Stereopsis Method for Visually Impaired to Identify Obstacles Based on Distance
,” Proceedings of the Third International Conference on Image and Graphics
, pp. 580
–583
.38.
Zitnick
, C.
, and Kanade
, T.
, 1999, “A Cooperative Algorithm for Stereo Matching and Occlusion Detection
,” Robotics Institute, Carnegie Mellon University, Technical Report No. CMU-RI-TR-99-35.39.
Roy
, S.
, and Cox
, I.
, 1998, “A Maximum-Flow Formulation of the N-Camera Stereo Correspondence Problem
,” Sixth International Conference on Computer Vision
, pp. 492
–502
.40.
Birchfield
, S.
, and Tomasi
, C.
, 1996, “Depth Discontinuities by Pixel-to-Pixel Stereo
,” Stanford University, Technical Report No. STAN-CSTR-96-1573, pp. 1073
–1083
.Copyright © 2009
by American Society of Mechanical Engineers
You do not currently have access to this content.
