Presentation
Towards User-Centered Robot Control to Support Individuals with Motor Impairments
SessionRobotics Summit - Session 4
DescriptionBackground | Globally, one out of 10 people has some form of disability. Approximately 39 million Americans have motor impairments. This impairment is the most common in the US, with 1 in 7 adults (about 13.7%) having trouble getting around, walking or climbing stairs. Technology can potentially support individuals with disabilities, their families, and professional caregivers with a broad range of capabilities. Assistive robotic technologies are rapidly developing as a feasible option for support. Some assistive robots can potentially support the autonomy and independence of people with motor impairments. However, most are not explicitly designed for this population and their needs, leading to challenges adapting these robots to suit their needs. One of the significant challenges is the high task load demand and the complications of controlling the robot to accomplish the tasks needed. The current user interfaces of assistive robots are very technically demanding, which often limits the independence and flexibility users (particularly people with motor impairments) have over the control of the robot. We aim to employ an iterative participatory design process to develop a novel and intuitive control mechanism to enable people with motor impairments to operate assistive robots to perform activities independently.
System Design | The system comprises an assistive Stretch robot (from Hello Robot Inc.) and a novel hands-free controller, Mouse Integrated Glasses (MIG from Hourglass Medical). The Stretch robot is a mobile robot manipulator designed to support everyday activities using a lightweight telescoping arm mounted on a mobile base. Stretch can be operated via a web-based user interface at three levels of robot autonomy - teleoperation (the user controls the robot’s actions), semi-autonomy (the user selects options, the robot executes the chosen options), and full automation (the robot performs all actions). We interfaced the MIG to the robot’s computer via Bluetooth. The MIG has an embedded gyroscope that senses the rotational motion of the user’s head and a jaw clench sensor. The MIG device can conveniently move a mouse cursor, generate button clicks, and drag and drop actions without additional hand controls. While connected to the Stretch robot, the users can perform tasks such as picking up and delivering an item and cleaning a surface. To do these actions, the user wears the glasses and makes head movements (such as nods and slight head turns) to move the robot. A light jaw clench generates a mouse click input, translated through the interface to carry out actions such as gripping or additional movements. It can be programmed to execute more actions through the robot.
Protocol | While designing the protocol for the participatory design, we prepared instructional protocols composed of instructional guides and a brief instructional video to support the user in the setup and use of the MIG to operate the robot. We also set up a computer-based mouse tracking and pointer accuracy and efficiency test to evaluate the users' abilities to use the MIG easily. This test captured the effectiveness, precision and accuracy of jaw clenching, head movements and mouse control. We intend to deploy these validated procedures for iteratively designing a suite of instructional materials to help users efficiently use the MIG to control Stretch and assist with tasks of their choice.
Feasibility Test | We have conducted internal feasibility tests where we tested this training protocol. These tests were conducted at the McKechnie Family LIFE Home (a home simulation environment focused on innovation in home environments). The test helped identify knowledge gaps or training challenges the users could encounter while preparing to use the MIG for the first time. All pilot users could use the MIG in the computer-based test and successfully control the robot. Users shared ideas on helpful content that could be included in the training materials to make it more user-friendly and support their introduction to the novel device. We also needed to adjust the sensor locations and make jaw clench adaptations to match our pilot users' head and jaw line configurations. These internal tests have yielded a clearer understanding of the task domain, haptic interface design requirements, usability challenges that need to be addressed, and control alternatives that would ease user control. We plan to discuss some of the design insights obtained from these internal tests, which we believe will benefit multiple stakeholders considering related designs or already involved in such designs to support people with motor impairments.
Next Steps | Further plans are in place to recruit participants who self-report upper and/or lower body motor impairments to test out the current version of the system and perform various household tasks. We intend to evaluate perceived usefulness, ease of use, and task load. This will be followed by a structured interview with questions to elicit potential use cases and control mode preferences. This participatory design approach will be conducted iteratively, yielding needs and the technical requirements to inform and advance the design of the hands-free controller while integrating user-centered autonomy and feedback to support the users. Alternative user input options will also be incorporated in later stages of the design, such as augmented reality (AR) through the MIG to control Stretch at different degrees of autonomy. Other input modalities such as voice, head nods, and blinks could also be incorporated iteratively along with AR based on the user needs identified. The mapping of input events to specific robot actions will be developed to different degrees of autonomy and transparency.
This design is expected to provide an intuitive interface for robot functions and autonomy integrations through formative human factors methods. Autonomous and remotely controlled functionalities will be seamlessly integrated with the hands-free controller in embedded evaluations in home environments to support people with motor impairments.
System Design | The system comprises an assistive Stretch robot (from Hello Robot Inc.) and a novel hands-free controller, Mouse Integrated Glasses (MIG from Hourglass Medical). The Stretch robot is a mobile robot manipulator designed to support everyday activities using a lightweight telescoping arm mounted on a mobile base. Stretch can be operated via a web-based user interface at three levels of robot autonomy - teleoperation (the user controls the robot’s actions), semi-autonomy (the user selects options, the robot executes the chosen options), and full automation (the robot performs all actions). We interfaced the MIG to the robot’s computer via Bluetooth. The MIG has an embedded gyroscope that senses the rotational motion of the user’s head and a jaw clench sensor. The MIG device can conveniently move a mouse cursor, generate button clicks, and drag and drop actions without additional hand controls. While connected to the Stretch robot, the users can perform tasks such as picking up and delivering an item and cleaning a surface. To do these actions, the user wears the glasses and makes head movements (such as nods and slight head turns) to move the robot. A light jaw clench generates a mouse click input, translated through the interface to carry out actions such as gripping or additional movements. It can be programmed to execute more actions through the robot.
Protocol | While designing the protocol for the participatory design, we prepared instructional protocols composed of instructional guides and a brief instructional video to support the user in the setup and use of the MIG to operate the robot. We also set up a computer-based mouse tracking and pointer accuracy and efficiency test to evaluate the users' abilities to use the MIG easily. This test captured the effectiveness, precision and accuracy of jaw clenching, head movements and mouse control. We intend to deploy these validated procedures for iteratively designing a suite of instructional materials to help users efficiently use the MIG to control Stretch and assist with tasks of their choice.
Feasibility Test | We have conducted internal feasibility tests where we tested this training protocol. These tests were conducted at the McKechnie Family LIFE Home (a home simulation environment focused on innovation in home environments). The test helped identify knowledge gaps or training challenges the users could encounter while preparing to use the MIG for the first time. All pilot users could use the MIG in the computer-based test and successfully control the robot. Users shared ideas on helpful content that could be included in the training materials to make it more user-friendly and support their introduction to the novel device. We also needed to adjust the sensor locations and make jaw clench adaptations to match our pilot users' head and jaw line configurations. These internal tests have yielded a clearer understanding of the task domain, haptic interface design requirements, usability challenges that need to be addressed, and control alternatives that would ease user control. We plan to discuss some of the design insights obtained from these internal tests, which we believe will benefit multiple stakeholders considering related designs or already involved in such designs to support people with motor impairments.
Next Steps | Further plans are in place to recruit participants who self-report upper and/or lower body motor impairments to test out the current version of the system and perform various household tasks. We intend to evaluate perceived usefulness, ease of use, and task load. This will be followed by a structured interview with questions to elicit potential use cases and control mode preferences. This participatory design approach will be conducted iteratively, yielding needs and the technical requirements to inform and advance the design of the hands-free controller while integrating user-centered autonomy and feedback to support the users. Alternative user input options will also be incorporated in later stages of the design, such as augmented reality (AR) through the MIG to control Stretch at different degrees of autonomy. Other input modalities such as voice, head nods, and blinks could also be incorporated iteratively along with AR based on the user needs identified. The mapping of input events to specific robot actions will be developed to different degrees of autonomy and transparency.
This design is expected to provide an intuitive interface for robot functions and autonomy integrations through formative human factors methods. Autonomous and remotely controlled functionalities will be seamlessly integrated with the hands-free controller in embedded evaluations in home environments to support people with motor impairments.
Event Type
Robotics Workshop Submission
TimeSunday, March 303:40pm - 4:00pm EDT
LocationHarbour A/B