Neptune: Mobile Manipulator With Advanced Human Robot Interaction

This thesis describes Neptune, a mobile manipulator designed as an assistive device for task-related activities and rehabilitation of children with special needs. Neptune consists of a mobile robot base and a 6DOF robotic arm, and it is interfaced to users via Wii Remote, iPad, Neuro headset, a camera, and pressure sensors. These interfaces allow patients, therapists and operators to interact with the robot in multiple ways, as may be appropriate in assistive scenarios such as: direct physical interaction with the iPad, arm positioning exercises through Wii remote, remote navigation and object retrieval through the environment via the Neuro headset, etc. In this thesis we present an overview of the system and discuss its future uses in rehabilitation of CP children. In this thesis, we have investigated 5 different modalities of interaction with robots. Motion sensing: we present a novel algorithm to map the Wii remote motions to the mobile manipulator. This enables the user to guide the robot in a natural way by pointing the remote in a specified manner.Physical sensing: we present a novel approach to enhance human robot interactivity through the use of force feedback with the force sensors. Visual sensing: we present a novel approach to enhance interaction of mobile manipulator through the use of visual servoing and tracking. Interface devices: we present work on combining dynamic gesture based commands from an interface device to improve the intuitiveness of control and planning of a multiple degrees of freedom robot system through neural network based learning.Neural Signal Sensing: The brain activity is read and feature extraction, classification algorithms are applied to detect cognitive, affective and expressive features of the person wearing the neuro headset. These features are mapped to operate the robot.We propose an efficient way to coordinate multiple modalities sensing as generalized interface for multiple robots. Performance metrics are proposed so that we have the quantitative way to identify our interaction efficiency.