Turning thought to action

A new approach to myoelectric prosthetics gives amputees unprecedented, intuitive control of a robotic hand

Turning thought to action

A new approach to myoelectric prosthetics gives amputees unprecedented, intuitive control of a robotic hand

Woman wearing a body-powered prosthetic right arm reaches for a piece of laundry with her left hand.
Karen Sussex of Jackson, Mich., folds and hangs laundry using her body-powered prosthesis — a limited, decades-old technology that many amputees still prefer due to its simplicity and reliability. Sussex is a participant in an advanced prosthetics study at the University of Michigan.

Abstract

Losing a limb can be a devastating setback that science and technology struggle to fully offset, even after thousands of years of effort. Prosthetic devices in use today offer limited functionality or can be too cumbersome for amputees to use effectively.

While advanced robotic hands exist, amputees don’t have ways to intuitively control them. That lack of naturalistic control contributes, in many cases, to some abandoning their prostheses because they find life easier without them.[1]

A biological interface developed by clinicians and engineers at the University of Michigan is giving amputees new, intuitive control of the most advanced robotic hands on the market. Their new approach centers on the Regenerative Peripheral Nerve Interface (RPNI)—a small graft of muscle tissue surgically attached to the end of a severed nerve in an amputee’s arm.

While other neural interfaces are harmful to nerves, the RPNI promotes healthy nerve growth and acts as a bioamplifier, converting faint neural signals sent from the brain into large, recordable muscle signals that remain stable for years. Combined with machine learning algorithms, these signals enable intuitive, real-time mind control of advanced robotic prosthetic hands.

The RPNI grafts also impede the growth of painful neuromas at the severed ends of nerves, reducing: the need for chronic pain medication, repeated surgical procedures, and an inability to use prosthetics due to pain levels.

This is a story about a research project at the University of Michigan.

The peer-reviewed paper was published on March 4, 2020 in the journal Science Translational Medicine and is titled, “A regenerative peripheral nerve interface allows real-time control of an artificial hand in upper limb amputees” DOI: 10.1126/scitranslmed.aay2857

The Regenerative Peripheral Nerve Interface, or RPNI, amplifies neural signals in the arm in order to be recorded and translated into control parameters for an advanced prosthetic hand.

Introduction

Connecting the mind to a robotic prosthetic hand

The human brain controls arm and hand movement using multiple peripheral nerves that run from the spinal cord to all of the muscles of the arm and hand. Impulses sent from the brain are carried down through the spinal cord and along the peripheral nerves to generate muscle contractions, which pull on the bones in the hand via attached tendons, generating movement.

When a surgeon amputates part of an upper limb, those peripheral nerves are severed. Figure 1.1 But research has shown that even decades after an amputation, the brain still sends electrical impulses along those nerves in an attempt to control the muscles in the phantom limb.[2]

These signals can be recorded and used to control a prosthetic hand. U-M’s approach is to attach small muscle grafts, called RPNIs, to the end of the severed nerves.[3] The grafts are stimulated by the neural impulses, generating an electric potential that an embedded electrode records and transmits to a robotic prosthesis.[4]

Figure 1.1
nerve ending

Preventing Pain

How RPNIs can prevent neuroma and phantom pain

Woman wearing a body-powered prosthetic right arm stands at a kitchen stove browning meatballs.
Sussex prepares a special lunch of spaghetti and meatballs in her Jackson home. She often chooses to make simpler meals because of her body-powered prosthesis.

The current state of prosthetic hands

The body-powered hook continues to be used by a large percentage of amputees due to ease of use and affordability. But its function is limited. It can only open and close and requires a harness that runs behind the back and over the opposite shoulder.

Myoelectric prostheses offer more functionality but the most common varieties available today fall short of an intuitive solution. These devices rely on electromyographic (EMG) signals from an amputee’s remaining muscles, typically recorded by electrodes placed on the skin of the arm. For example, a person amputated above the elbow could flex their biceps and triceps to toggle between and activate different hand functions, including open/close fist, rotate wrist, and pinch index finger and thumb.

New, intuitive prosthetic systems can tap into the brain, nerves or muscle tissue to let an amputee simply think about moving their hand and the prosthesis responds as intended. U-M’s approach uses grafted muscle tissue to harness these signals in a way that is less invasive and longer lasting that the other approaches.

A surgeon in an operating room sutures the end of an amputated arm.
Daniel Lyons, a resident of plastic surgery at the University of Michigan, stitches up one side of Sussex’s arm after implanting three bipolar electrodes. The surgery team implanted a total of eight, and together they will help her intuitively control a prosthetic hand.

Toward intuitive prostheses

Researchers around the world are searching for ways to access and amplify the body’s neural commands to give amputees a more natural experience with a prosthesis.

Targeted muscle reinnervation (TMR), an advanced myoelectric approach, involves rerouting severed peripheral nerves to large nearby muscle groups and picking up their signals with external electrodes placed on the skin.[7] With TMR, an amputee can more naturally control a prosthetic hand because nerves that originally ran to the phantom hand now run to the chest, for example, and generate prosthetic-controlling muscle contractions there. The electrodes that pick up these contractions can shift and encounter sweat, though, making TMR susceptible to signal disruption. It also requires an invasive surgical procedure and offers only a limited number of usable signals.

Epineural electrodes, often called “cuff” electrodes, wrap around the outside of a nerve and record the electric potential near its surface.[8] Without accessing the center of the nerve bundle, the full signal is not captured. In addition, nerves don’t like being touched by foreign material, leading to irritation, scar tissue and a signal-to-noise ratio that weakens over time.

Intraneural electrodes are tiny needles that penetrate the nerve itself, allowing for more selectivity in which signal is captured.[9] It’s an approach that has yielded encouraging results, but scar tissue can form, diminishing signal strength.

U-M’s RPNIFigure 1.2, another advanced myoelectric approach, offers a stable method for recording large signals.[10] Nerves stay healthier because they aren’t being re-routed or touched by electrodes. Instead, they’re growing into natural muscle tissue, which amplifies the nerve signals. Indwelling electrodes record those signals. Surgeons can easily learn the RPNI procedure, and patients can receive it during an amputation or years later.

Figure 1.2
nerve ending wrapped in muscle tissue

The evolution of the RPNI

A previous approach that was considered too complicated

The procedure for creating a Regenerative Peripheral Nerve Interface, or RPNI, consists of making a clean cut at the end of a nerve, suturing it to a small muscle graft, wrapping the graft around the nerve and stitching it shut.

Methods

The researchers partnered with upper-limb amputees and used ultrasound, electromyogram (EMG) recording and both virtual and physical prosthetic control tasks to validate whether human amputees can use RPNIs to naturally control an advanced robotic hand over a long-term period.

Three participants had elected to undergo RPNI surgery at U-M in order to treat pain, and they later agreed to join this prosthetics control study.Figure 2.1

Figure 2.1
human body outlines representing three participants

Participant Details

Study participants, by the numbers

Creation of the Regenerative Peripheral Nerve Interface, or RPNI

To create each RPNI, surgeons harvested a 3 x 1.5 x 0.5 cm graft from a healthy muscle elsewhere on the patient’s body, often the vastus lateralis (large thigh muscle). Then they made a clean cut on the end of a peripheral nerve in the amputated limb to remove the neuroma. If multiple RPNIs were planned for that peripheral nerve, surgeons divided it into individual nerve bundles (fascicles). Then they wrapped the muscle graft around the end of the nerve and sutured it in place. They repeated the process until they created the desired number of RPNIs.

In the three months following surgery, the participant’s body helped the RPNI regenerate, revascularize and reinnervate.

A researcher applies an ultrasound wand near the end of a study participant’s amputated arm.
Ann Laidlaw, an assistant professor of physical medicine and rehabilitation, and Philip Vu, a biomedical engineering PhD candidate, use ultrasound in a University of Michigan lab to locate and image eight electrodes recently implanted in Karen Sussex’s arm. This was Sussex’s first lab session after surgery to implant the electrodes.

Observing RPNI contractions via ultrasound imaging

Post-surgery, the team utilized ultrasound to visually measure the physical contractions of the RPNIs as each participant thought about moving their phantom hand in specific ways. The researchers quantified the strength of the RPNI contraction by measuring the amount of pixel change in an ultrasound video clip between an RPNI at rest and at maximum contraction.

Recording EMG signals from RPNI using indwelling electrodes

Their next step was recording EMG signals within the RPNIs as the participants thought about and generated specific phantom hand movements. An EMG signal is a measurement of the electric potential that muscle tissue generates when stimulated by a nerve.

Researchers first measured EMG signals by temporarily inserting, via a needle, thin wires through the skin and into the RPNIs. They removed the wires after each recording session. All three amputees participated.

Later, Hamilton and Sussex consented to surgeons implanting electrodes and wires for a specified length of time. Figure 2.2 Surgeons placed clinical, bipolar electrodes into each RPNI and ran wires through their limbs and out through a port. Hamilton had his removed after one year.

In both temporary and more permanent electrode configurations, researchers had the participants think about making specific postures or movements with their phantom hands. The team measured the strength of EMG signals from each electrode and how signals from different electrodes correlated with different hand gestures or finger movements.

Figure 2.2
hands holding surgical instruments, working around an open incision

Incorporating Electrodes

Electrode implantation photo gallery

WARNING: Graphic content.

Study participant watching a laptop screen while using  an advanced prosthetic hand to mimic hand positions shown on the screen.
Hamilton performs a discrete mirroring task as a participant in a University of Michigan study centering on the Regenerative Peripheral Nerve Interface, or RPNI. When the avatar hand on the screen cues a posture, Hamilton must think about matching it with his phantom hand to control the prosthetic hand and complete the task.

Using RPNI to intuitively control a prosthetic hand

Once the team was able to record EMG signals from the RPNIs, they had participants try to intuitively control both virtual and physical prosthetic hands. Two varieties of machine learning algorithms built by the researchers took in recorded EMG signals and generated control parameters for the prosthetic hands.

The first algorithm used a naive-Bayes classifierFigure 2.3 to recognize patterns in EMG signals and pair patterns with different hand postures. Participants spent only a few minutes training the algorithm—using their phantom hand to mirror virtual avatar hand postures such as fist, point, pinch, etc. After the short training session, the program immediately compiled the data and generated control parameters.

In one “discrete” task, the participant controlled a virtual or physical prosthetic hand in real time to mirror the posture of a separate virtual avatar hand. The researchers measured whether the participant was successful, and how quickly.

In another task, the participant controlled a physical prosthetic hand mounted to their amputated limb to perform simple tasks, like holding or moving objects of different shapes and sizes.

In a third exercise, researchers played a game of rock, paper, pliers with the participant. Pliers, or pinch, was a substitute for the traditional scissors, since a scissor motion could not be performed mechanically by the prosthesis.

The second type of algorithm used a Kalman filterFigure 2.3 to estimate the intended position and velocity of specific fingers given a series of observations. During the short training phase, participants watched a virtual avatar hand move its thumb and mimicked it with their phantom hand. The algorithm would ingest each participant’s EMG data as well as the position and velocity of the avatar hand in order to recursively generate an improving control algorithm until the training was complete.

In one “continuous” task, called a center-out task, participants controlled the thumb of a virtual or physical prosthetic hand in real time to hit various targets. Researchers measured whether each participant was successful, and how quickly.

In another task, participants controlled a physical prosthetic hand attached to their amputated limbs to perform slightly more complex tasks, such as stacking small blocks, using a key, and using a zipper.

Figure 2.3
robotic hand demonstrating thumb movement

Machine Learning

Adapting and fine-tuning control algorithms

Study participant sticking out his tongue in concentration while using an advanced prosthetic hand to move a zipper.
Joe Hamilton controls the thumb and index finger of the DEKA prosthetic hand in order to open a zipper. Hamilton is participating in a study at the University of Michigan that centers on the Regenerative Peripheral Nerve Interface, or RPNI.

Results

RPNIs create natural prosthetic control with long-term stability

Using ultrasound imaging, EMG recording and prosthetic control tasks, researchers demonstrated that RPNIs attached to the arm’s peripheral nerves will physically contract when an amputee thinks about moving their phantom hand. This generates large EMG signals to serve as real-time control signals for a robotic prosthetic hand. The RPNIs remain stable for years and offer greater levels of control than other approaches.

Strong RPNI contractions observed using ultrasound

Ultrasound testing showed clear contractions of the RPNIs when participants thought about moving their phantom hand. Figure 3.1Specific RPNIs fired, either individually or in combination with others, as expected based on the hand gestures being performed. And within individual RPNIs, researchers saw sub-regions responding uniquely to the tasks performed, revealing a depth to the data present in the grafts.

Figure 3.1
person performing ultrasound on instrumented arm

Ultrasound of RPNI Contractions

Live ultrasound of RPNI muscle grafts contracting.

Watch Video Video with audio descriptions

Large signal-to-noise ratio recorded from RPNIs

The mean signal-to-noise (SNR) ratios recorded during testing were 4.21 for Moran, 68.9 for Hamilton, and 21.0 for Sussex. Figure 3.2 The range is likely large for several reasons. Researchers obtained Moran's signal using 1 mm fine wire electrodes. They obtained signals from Hamilton and Sussex via larger, 10 mm electrodes. Between the latter two, Hamilton’s RPNIs had been in place for two years longer than those of Sussex prior to electrode implantation, giving more time for nerves to regenerate and the RPNIs to strengthen.

With other neural interface approaches, epineural electrodes and intraneural probes typically record SNRs between 4 and 15.[8] [9] [17] [18] One intraneural probe, the Utah Slanted Electrode Array, has been shown to record larger signals, but it currently only remains stable for months.[19] RPNIs, however, have remained stable for years.

Figure 3.2
electromyograph showing spikes and drops in activity

Stronger Signals

Live electromyographic (EMG) signals during a testing session

Watch Video Video with audio descriptions
Study participant using an advanced prosthetic arm to reach for a can of tomato paste
Sussex controls a Touch Bionics i-LIMB prosthetic hand to pick up a can of tomato paste. Photo: Robert Coelius

Quick and dexterous control of prosthetic hand using RPNIs

Hamilton and Sussex successfully controlled a prosthetic hand to mirror gestures and movement of both a virtual and physical hand, as well as manipulated objects and played games like rock, paper, pliers. Figure 3.3

In a posture-mirroring "discrete" task, the two participants controlled a virtual hand to intuitively generate and switch between useful hand postures like pinch, fist and point. They matched postures quickly with a success rate between 94% and 100%. The average time between an RPNI contraction and the team’s control algorithm generating a command was less than 0.3 seconds.

In the center-out "continuous" movement task, participants intuitively moved their thumb along two degrees of freedom—forward and back and side to side—giving them the ability to move their thumb in a circle. When testing only the forward and back motion, both participants hit between 96% and 100% of the targets on the first day. Without recalibrating the control algorithms, Hamilton still hit 100% on day 300. Sussex hit 96.4% on day 97, her last recorded day for this study.

Figure 3.3
human in fist, robotic hand flat and extended

Intuitive Control

Performing tasks with virtual and physical prostheses.

Watch Video Video with audio descriptions
Close up of advanced prosthetic hand with a green block grasped between the thumb and index finger, positioning the block at the top of a stack.
Joe Hamilton intuitively controls a DEKA hand to gently stacks a set of small blocks using the University of Michigan’s RPNI platform.

Conclusion

Advanced prosthetic control via RPNI and the way forward

The University of Michigan's RPNI approach shows great promise for improving quality of life for amputees. It can effectively treat pain and enable more intuitive, life-like use of prosthetic devices.

These small pieces of grafted muscle naturally interact with the body’s nervous system to amplify neural impulses, delivering strong, clear signals for naturalistic, real-time prosthetic control. And these signals remain stable over the long-term, beyond what other approaches can offer.

The team’s current FDA Investigational Device Exemption allows them to implant twelve indwelling electrodes in each of 10 participants. They are looking for more study participants to partner with, especially those with upper arm amputations.

Beyond this 10-participant study, the team is evaluating the use of finer electrodes, and more of them, for greater EMG signal resolution. Future enhancements could include adding a wireless capability and exploring the effectiveness of RPNIs for prosthetic leg and foot control. The U-M team has also seen evidence that the RPNI can send sensory information back to the brain—a focus of further study.

In addition, RPNIs generate control signals for finer movements of the hand than what most robotic prostheses can perform at present. The team has begun collaborating with prosthetic device companies to help future devices accommodate these finer movements.