Researchers from École Polytechnique Fédérale de Lausanne (EPFL) in Switzerland and the University of Texas at Austin (UT) have developed a brain-computer interface (BCI) that allows users to modify a robot manipulator's motion trajectories. The system uses inverse reinforcement learning (IRL) and can learn a user's preferences from fewer than five demonstrations.
The system and a set of experiments were described in a paper published in Nature's Communications Biology journal. The goal of the research was to help paralyzed patients by developing assistive robots that can be controlled using a BCI. The team recorded electroencephalogram (EEG) and electrooculogram (EOG) signals from users while they commanded a robot manipulator with a joystick. The robot's software included a semi-autonomous obstacle avoidance routine. The routine's parameters were updated using IRL based on error-related potentials: signals decoded from the EEG and EOG signals, whereby the robot learned how close to a fragile obstacle it could move without making the user uncomfortable. According to lead researcher Aude Billard,
Assistance from robots could help [people with a spinal cord injury] recover some of their lost dexterity, since the robot can execute tasks in their place.
BCI devices typically measure neural activity using internal implants or external sensors such as EEG electrodes. The goal is to convert this sensor data into a signal that can be used as a computer input, so that users may control a computer or robot to do tasks that the user is physically unable to do. This often requires the user to imagine performing a physical activity, which results in neural activity that the BCI can detect and convert to a computer input.
Because directly commanding a robot manipulator via a BCI could be time-consuming and fatiguing, as the user would need to constantly be "in control," Billard's team chose to investigate how a BCI could be used to adjust the behavior of a semi-autonomous robot manipulator. In particular, the system adjusts the robot's obstacle avoidance algorithm in response to the user's error-related potentials (ErrP): EEG signals that are detected and decoded when the robot moves closer to an obstacle than the user expects.
To make this adjustment, the researchers implemented an IRL training algorithm. Unlike typical RL algorithms, where a learning agent's behavior is scored by a reward function, IRL learns both the reward function and the optimal action from a set of demonstrations. The demonstrations were performed by users directing a robot manipulator via a joystick to move either right or left in a workspace that contained a fragile obstacle. As the manipulator approached the obstacle, the robot would attempt to avoid the obstacle; if the user anticipated that the robot would not avoid the obstacle, the ErrP signal detected by the BCI was used to adjust the reward function and obstacle avoidance parameters.
In a set of experiments, the researchers found that their system could identify a user's reward function in as few as three demonstrations. They also noted that their approach was "robust to the natural variability and sub-optimal performance of the ErrP decoder," a useful property, as EEG sensing can be noisy.
The use of machine learning techniques for improving BCI devices is an active research area. In 2019, InfoQ reported on several efforts from university researchers to use brain signals to synthesize speech as well as Facebook Reality Lab's device that allows users to "type" by imagining themselves speaking. In 2021, InfoQ covered a wireless BCI which allowed users to control a video game.
The robot obstacle avoidance and IRL code used by Billard's team is available on GitHub.