Unmanned ground vehicles have the potential for supporting small dismounted teams in mapping facilities, maintaining
security in cleared buildings, and extending the team’s reconnaissance and persistent surveillance capability. In order
for such autonomous systems to integrate with the team, we must move beyond current interaction methods using heads-down
teleoperation which require intensive human attention and affect the human operator’s ability to maintain local
situational awareness and ensure their own safety.
This paper focuses on the design, development and demonstration of a multimodal interaction system that incorporates
naturalistic human gestures, voice commands, and a tablet interface. By providing multiple, partially redundant
interaction modes, our system degrades gracefully in complex environments and enables the human operator to robustly
select the most suitable interaction method given the situational demands. For instance, the human can silently use arm
and hand gestures for commanding a team of robots when it is important to maintain stealth. The tablet interface
provides an overhead situational map allowing waypoint-based navigation for multiple ground robots in beyond-line-of-sight
conditions. Using lightweight, wearable motion sensing hardware either worn comfortably beneath the operator’s
clothing or integrated within their uniform, our non-vision-based approach enables an accurate, continuous gesture
recognition capability without line-of-sight constraints. To reduce the training necessary to operate the system, we
designed the interactions around familiar arm and hand gestures.