Research ArticleBioengineering

Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses

See allHide authors and affiliations

Science Translational Medicine  05 Feb 2014:
Vol. 6, Issue 222, pp. 222ra19
DOI: 10.1126/scitranslmed.3006820

Abstract

Hand loss is a highly disabling event that markedly affects the quality of life. To achieve a close to natural replacement for the lost hand, the user should be provided with the rich sensations that we naturally perceive when grasping or manipulating an object. Ideal bidirectional hand prostheses should involve both a reliable decoding of the user’s intentions and the delivery of nearly “natural” sensory feedback through remnant afferent pathways, simultaneously and in real time. However, current hand prostheses fail to achieve these requirements, particularly because they lack any sensory feedback. We show that by stimulating the median and ulnar nerve fascicles using transversal multichannel intrafascicular electrodes, according to the information provided by the artificial sensors from a hand prosthesis, physiologically appropriate (near-natural) sensory information can be provided to an amputee during the real-time decoding of different grasping tasks to control a dexterous hand prosthesis. This feedback enabled the participant to effectively modulate the grasping force of the prosthesis with no visual or auditory feedback. Three different force levels were distinguished and consistently used by the subject. The results also demonstrate that a high complexity of perception can be obtained, allowing the subject to identify the stiffness and shape of three different objects by exploiting different characteristics of the elicited sensations. This approach could improve the efficacy and “life-like” quality of hand prostheses, resulting in a keystone strategy for the near-natural replacement of missing hands.

INTRODUCTION

Sophisticated hand control is a peculiar characteristic of higher primates. Dexterous manipulation is achieved through a complex relationship between motor commands, executed movements, and sensory feedback during hand activities. Hand loss causes severe physical debilitation and often distress because skillful object grasping and manipulation are compromised, thus depriving the person of the most immediate and important source of tactile sensing in the body. For these reasons, replacing a lost hand and its precise functionalities is a major unmet clinical need that is receiving attention from engineers, neurophysiologists, and clinicians. An ideal hand prosthesis should reproduce the bidirectional link between the user’s nervous system and the peri-personal environment by exploiting the post-amputation persistence of the central and peripheral neural networks and pathways devoted to hand motor control (1) and sensing (25). In particular, real-time and natural feedback from the hand prosthesis to the user is essential to enhance the control and the functional impact of prosthetic hands in daily activities, prompting their full acceptance by users within an appropriate “body scheme” that does not require continuous visual monitoring, as with current artificial hands (6, 7).

Recent notable advances in the field of hand prostheses have included designing devices with multiple degrees of freedom and equipped with different sensors (810). These developments have made the need for more effective bidirectional control even more compelling. A promising solution is represented by targeted muscle reinnervation (TMR), which consists of rerouting the residual nerves of the amputees over the chest muscles (11, 12). Individuals with arm or hand amputations can chronically use TMR-based prostheses, which could theoretically allow for a certain amount of sensory feedback (13, 14). However, because the superficial electromyogram (sEMG), used as a control signal, is recorded from the same body region (that is, the chest) that must be mechanically stimulated to provide feedback, real-time bidirectional control could be difficult to achieve. In this scenario, TMR subjects must contract muscles and simultaneously perceive a touch sensation on the skin overlying the same muscles, therefore possibly producing the so-called neurophysiological “sensory gating” (15).

In parallel, the rapid development of neural interfaces for the peripheral nervous system (16) has provided potential for new tools through which bidirectional communication with nerves in the stump could be potentially restored. Initial feasibility demonstrations of the induction of some sensations (17) and preliminary trials of the sporadic control of nonattached prostheses (1820) have recently been performed. However, to date, no evidence has been gathered for the real-time use of these neural interfaces for the effective bidirectional control of dexterous prosthetic hands performing different grasping tasks.

In this case study, our aim was to restore touch sensation in a person with hand amputation using transversal intrafascicular multichannel electrodes (TIMEs) (21) connected to artificial hand sensors and to intuitively use this sensation to achieve bidirectional control of a hand prosthesis. Our hypothesis was that the participant would be able to exploit the dynamic tactile information induced by neural stimulation that is triggered by the sensors of the hand prosthesis, during real-time simultaneous control of a dexterous prosthetic hand, to adaptively modulate grasping force, thus closing the user-prosthesis loop. The active sites of the electrodes were used to deliver electrical stimuli to the peripheral nerves that were proportional to the readouts of artificial sensors in the hand prosthesis. sEMG signals were used to decode different grasping tasks to verify whether the sensory information provided could be used in real time (that is, with the delay imperceptible by the user). If this goal could be accomplished, then the hand prosthesis could have practical relevance in daily activities.

RESULTS

TIMEs were implanted into the median and ulnar nerves of an amputee’s residuum to investigate the possibility to restore natural sensory feedback and integrate it into the user control loop of a dexterous prosthetic hand (Fig. 1A). Median and ulnar nerves (Fig. 1, B and C) were chosen because their innervation territory covers almost entirely the palmar and the finger sensory fields. The participant (D.A.S.) suffered a transradial left arm amputation 10 years ago, as a consequence of a traumatic event. During all experiments with restored touch, he was acoustically and visually isolated (Fig. 1A) to show that he was relying exclusively on the induced tactile sensation, without using other sensory modalities. The motor commands (palmar grasp, ulnar grasp, tridigital grasp, hand opening, and rest) were decoded by processing sEMG signals, whereas the sensory feedback (encoding) was triggered by intrafascicular stimulation of the median and ulnar nerves by using TIMEs (Fig. 1, D and E) according the information of the hand prostheses sensors. The subject performed more than 700 trials to verify his ability to modulate force during grasping and to identify specific physical characteristics of objects.

Fig. 1. Bidirectional control of hand prosthesis and characterization of neural stimulation.

During experiments, the participant was blindfolded and acoustically shielded. The real-time bidirectional multiple-grasp control of the hand prosthesis involved both a reliable decoding of the user’s motor command—immediately converted into hand motion (control loop)—and a simultaneous readout from prosthesis sensors fed back to the user through intrafascicular nerve stimulation (sensory loop). The decoding was performed by processing sEMG signals, whereas the encoding was simultaneously achieved by intrafascicular stimulation of the median and ulnar nerves using TIMEs. (A) The current was delivered as a function of the prosthetic hand sensor readouts. S15 and S75 are 15 and 75% of the range of sensor values, respectively. (B) Photograph of the surgical insertion of a TIME electrode in the median nerve of the participant. (C) Depiction of the subject’s ulnar nerve with the two implanted electrodes. (D) Time course of the reported threshold and saturation of sensation over 4 weeks in the little and index fingers. The sensation threshold corresponded to the minimal sensation of touch reported, whereas saturation (“pain threshold”) was defined as the charge that elicited a nearly painful touch as reported by the subject. (E) Sensation strength for each finger [color-coded as in (D)] reported on a scale from 1 to 10 for each of the 4 weeks.

Peripheral nerve stimulation

Sensory tactile feedback was restored by delivering electrical current through one TIME active site placed in the median nerve and one placed in the ulnar nerve. The sensations elicited corresponded to the physiological sensory mapping of touch within the innervation territories of the median and ulnar nerves (Fig. 1D), confirming that this property was not lost several years after sensory deprivation.

Then, we determined the range of electric charge usable (Fig. 1D) to provide a dynamically graded sensory feedback for real-time control. The lower threshold corresponded to the minimum stimulus charge needed to elicit the first distinct touch sensation. Saturation was positioned just below the pain threshold reported by the participant. When repeatedly tested, the mapping of sensations over the representation of the missing hand, assessed using a patient report, was stable and repeatable throughout the 4 weeks of study for both the median and the ulnar nerves (Fig. 1E). Stimulation of fascicles inside the ulnar nerve produced the sensation of gradual touch in the little finger, whereas stimulation inside the median nerve produced progressive sensations located in both the index finger and the thumb.

The charge range for the ulnar nerve was stable over 4 weeks, whereas for the median nerve, the charge range increased over time (Fig. 1D). Meanwhile, the maximum charge injected into both nerves (8 and 24 nC) was much lower than the highest charge theoretically injectable using TIME electrodes (120 nC). The injected charge range used in this study was consistent with the range 0.3 to 60 nC used by Dhillon et al. (17) for the stimulation of the human median and ulnar nerves with intrafascicular electrodes.

Real-time fine force control

We first assessed whether the induced natural (physiologically plausible) dynamic sensory feedback could lead to a voluntary and reliable modulation of grasping force of the prosthesis. During these fine force trials (n = 449), the participant was blindfolded and acoustically shielded (Fig. 1A). The subject was asked to repeatedly produce three different force levels on a pressure sensor chamber, using the induced sensory feedback to modulate the force when performing pinch, ulnar, and palmar grasps.

Assessment of the performance during closed-loop grasping control

The participant was able to accurately control the grasp force both in single-level press-and-release trials and in the continuously modulated “staircase task” (Fig. 2A). In the staircase task, the participant needed to gradually increase the applied force in three steps lasting about 2 s each and then gradually return to the baseline, whereas in the single-level trials, he was instructed to maintain it until feeling confident with the exerted level of force. In certain trials, when the participant recognized—through the sensory feedback—that he had applied too much pressure (Fig. 2A, red arrow), he corrected the grasp accordingly, eventually meeting the task requirement and clearly exploiting the induced sensory feedback.

Fig. 2. Fine force control.

(A) External pressure sensor outputs (black) normalized to 1.5 kPa, hand sensor readings (blue) normalized to 60 N, and stimulation current amplitude (red) normalized to 240 and 160 μA for the index and little fingers, respectively, during force control task for the index and little fingers. A red arrow indicates when the subject recognized an overshoot in the exerted pressure. Data are representative of 200 trials. (B) Confusion matrices of the requested versus exerted force levels for the index (n = 128 repetitions in seven sessions) and little fingers (n = 72 repetitions in four sessions). On the right, data are presented as the overall performance improvements during the experiment time course. (C) Comparison among the performance of the prosthetic hand without induced tactile feedback (visual only), with induced tactile feedback (no visual), and the healthy hand (n = 21). Time evolution and PC analysis are presented. The ellipsoids represent the location of each data group in the PC space (center, mean; semi-axis, SD). (D) Confusion matrix for force control task with a palmar grasp (n = 111 repetitions in two sessions). In matrices in (B) and (D): l, low; m, medium; h, high force levels.

Over the course of 7 days, the subject could produce three levels of stable and discrete pressure with both the index and the little fingers under voluntary control, reaching a success rate of >90% in some of the last sessions for both fingers (Fig. 2B). The accuracy of execution increased from the first day until the end of the experiment. The percentage of correct performance increased from an initial 67 to 93% for the index finger and from 56 to 83% for the little finger, demonstrating that the subject had most likely undergone a learning process by integrating the restored sensation into closed-loop control strategies (Fig. 2B). For this one subject, the predominant source of error was confined to the execution of the medium level of force (as reflected in Fig. 2B confusion matrices). With both index (fig. S1A) and little fingers (fig. S1B), the subject was able to apply consistently three significantly different force levels [Tukey-Kramer test, P < 0.001, single press-and-release trials (n = 76)].

To compare the performance of the hand prosthesis to that of an intact hand, the participant was asked to perform the same task with his intact (and dominant) right hand and with his left hand prosthesis with visual and acoustic feedbacks, but without any nerve stimulation. The obtained pressure profiles were evaluated by means of several parameters defining the precision and shape of the staircase (a list of all the parameters is provided in fig. S2). All data were processed using principal components analysis (PCA) and plotted in the space of the first three principal components, which explained more than 80% of the variance. Each ellipsoid represents one distribution, and it is centered at the mean value with semi-axes equal to the SD along each principal component (Fig. 2C). Trials executed solely with the artificial tactile feedback (in blue) were much more similar to those of the intact hand (in green) than the ones with prosthesis control by visual feedback (in red). The data variability in the absence of tactile feedback was greater than for the other two cases, as shown in the PCA representation in Fig. 2C, because the SDs for the case without tactile feedback are about double than in the case of tactile feedback induction. The performance with visual-only feedback was also significantly different from the one with the natural hand (Kruskal-Wallis, P < 0.05), whereas the performance of the natural hand versus the prosthetic hand with tactile feedback was similar (Kruskal-Wallis, P = 0.31) (Fig. 2C).

We also investigated whether the participant was able to integrate multiple and independent pressure sensory inputs when performing a palmar grasp (which delivered inputs from the median and ulnar region simultaneously). The results indicate that the dynamic information of the index and little fingers were effectively integrated and exploited to modulate palmar grasp fine force control, with 82.7% overall accuracy (Fig. 2D).

Falsification experiments

To evaluate whether the control of the exerted force was only due to the restored touch sensations over phantom hand fingers, and to falsify the hypothesis that the subject learned to regulate the force applied by modulating the timing of hand actuation, we designed and tested two different control conditions. In one test (placebo or “p”), among many trials with stimulation, we randomly selected several press-and-release trials in which the nerve stimulation was switched off while asking the patient to reach a low level of force; no force control was possible in this configuration (“Low-p” in Fig. 3A), resulting in the maximum possible force in all these trials (n = 15). In another test, the velocity of hand actuation was modified without notifying the subject to falsify the hypothesis that the user could still learn the force level exerted from the prosthesis closure time.

Fig. 3. Fine bidirectional control is due to integration of the restored sensation into the bidirectional loop.

(A) This test was similar to Fig. 2A except with placebo trials. Examples of placebo trials, randomly mixed during the sessions of the fine force protocol, are shown in “Low-p.” During these trials, the participant was asked to apply the minimum level of force but with the electrical stimulation turned off (no feedback from the sensors of the hand prosthesis). External pressure sensor outputs (black) normalized to 1.5 kPa, hand sensor readings (blue) normalized to 60 N, and stimulation current amplitude (red) normalized to 240 and 160 μA for the index and little fingers, respectively, during force control task for the index and little fingers. (B) Confusion matrices of the requested versus performed force levels for the index and little fingers at different velocities. The performance at different velocities of the robotic hand motors actuation was evaluated to exclude the possibility that the participant could have learned to control the force by associating the control with the time needed for hand prosthesis closure (n = 294 in seven sessions for the index finger and n = 155 in four sessions for the little finger). In matrices: l, low; m, medium; h, high force levels.

The achieved performances in Fig. 3B suggest that the participant was not relying on the timings of closure to reach a desired level of pressure. If this was the case, then both the slower and faster velocities should have resulted in poor or at least reduced performances compared with the normal one. Instead, at higher velocities, the subject had more difficulty in finely controlling the movements (with performances of 75.7% for the index finger and 67.0% for the little finger over all 131 sessions), whereas at lower velocities, the participant was able to finely grade the force, showing the best performance (88 and 94% performances for index and little fingers, respectively, over all 118 sessions). We speculate that this is because, at lower actuation speeds, it was easier to grade the exerted force and understand the intensity of the sensation. These results indicate that the force control exclusively relied on the induced sensory tactile feedback in this subject.

Functional grasping

We next investigated the possibility of integrating the restored sensation in a manipulation task similar to activities of daily living. During this task, an object was placed on the palm of the hand prosthesis, and the participant was instructed to recognize its position with respect to the hand and to perform the most appropriate grasp for handling. Three objects were placed at different locations: a cylindrical object, engaging the entire palm, and two smaller objects located on the median or ulnar sides. To recognize the object position, the participant, who was blindfolded and acoustically shielded, used an explorative palmar grasp (sensing phase in Fig. 4A). The real-time muscle electrical activity (sEMG envelope), the decoded hand commands, the synchronous readouts of the hand sensors, and the current injected within nerves (tactile traces) are represented in Fig. 4A. Once the position of the object was identified using the sensory information available, the participant reopened the hand and performed a location-appropriate grasp (palmar for the cylindrical object, and pinch or ulnar grasp for the median- and ulnar-located small object–specific grasp phases in Fig. 4A).

Fig. 4. Object sensing and selection of the appropriate grasp for handling.

(A) Three task repetitions involving a different palmar location of the object. Sensing phase: The participant performed a palmar grasp to detect the object position, then released the grip. Specific grasp phase (shaded): The participant performed the appropriate grasp for handling the item. The object was then displaced with a translation movement by the arm and released. sEMG signals at five sites (ExDC, ExCRL, FxCRp, FxCRd, and FxCU) were recorded and processed to decode the user’s hand motor commands, which drove the opening/closing of the hand prosthesis. The sensory feedback, encoded in terms of the intensity of intrafascicular nerve stimulation, arithmetically depended on the finger sensor traces (tactile traces). (B) Confusion matrix indicating a 97.3% mean class accuracy for selected grasps (n = 52). Main diagonal is accuracy for each class. pi, pinch; pa, palmar; ul, ulnar.

As soon as the participant realized that the object was steadily grasped, he handed the object to the experimenter sitting in front of him (in case of median position), to the experimenter on his right (for ulnar position), or lifted it up (for whole-palm sensing). The participant was able to perform this task with an average accuracy of ~97% over 52 trials performed on days 5 to 7 (Fig. 4B and movie S1).

Sensing the environment: Recognition of object properties

In this experiment, the participant’s ability to use the restored hand sensation to identify the physical properties of an object, such as stiffness and shape, was verified. In case of stiffness, the underlying hypothesis was that dynamics of the restored sensation could help discriminate specific object characteristics. Such discrimination would be based on the subject’s cognitive ability to decode stiffness by processing the artificial tactile feedback dispatched from the sensors of the prosthetic hand. In particular, because the stimulation currents injected into the nerves follow the sensor readout, it was expected to change very rapidly with a hard object and very slowly with a soft object.

The participant, blindfolded and acoustically shielded, was provided with three objects: a piece of wood (hard), a stack of plastic glasses (medium), and a cotton pack (soft). He was instructed to explore the object with a palmar grasp and to apply force until he could perceive its stiffness. As expected, the sensor readout in the hand significantly differed according to the object’s stiffness: Stiffer objects caused the sensor output (blue traces in Fig. 5A) and, consequently, the intensity of the injected current (red traces in Fig. 5A) to cross the range from minimum contact to saturation quicker than softer objects. The value of the average current amplitude derivative (0.67 ± 0.31, 0.19 ± 0.09, and 0.08 ± 0.04 μA/s for high, medium, and low stiffness, respectively) was significantly different for different object stiffnesses (Tukey-Kramer test, P < 0.001) (Fig. 5B and fig. S3). The evolution over time of the induced sensory response was consistently well separated, suggesting that this dynamic information could have been used by the subject to distinguish among the three presented stiffness (fig. S3).

Fig. 5. Object stiffness and shape recognition.

(A and B) Stiffness recognition, discrimination, and accuracy tasks. (A) Hand control decoded from sEMG activity and index finger robotic hand sensor readout and stimulation current amplitude, with its positive time derivative. (B) Confusion matrix and performance quantified for each of three sessions (n = 66) and current time derivative (positive, dI/dt) for the three objects. Data are averages ± SD. P values were determined by Kruskal-Wallis with Tukey-Kramer test for multigroup comparison. (C and D) Shape recognition, discrimination, and accuracy analysis. (C) Decoded motor commands, sensor readouts, and stimulation amplitude. The solid vertical lines and the gray double arrow indicate the stimulation onset delay between the median and ulnar sites. (D) Confusion matrix assessing task accuracy, based on the shape of the sensed object reported by the subject, and quantification of performance for each of three sessions (n = 32 total). The contact delay between the index and little fingers during manipulation differed between objects that engaged the full hand or partial hand. Data are averages ± SD. P values were determined by Kruskal-Wallis test.

It is reasonable to hypothesize that the subject might have intuitively exploited this information to distinguish between the three levels of stiffness within 3 s. As a result, the subject was able to consistently recognize the proposed object with an overall level of performance of 78.7% (Fig. 5B). A clear improvement was observed in intrasession performance, which might suggest a learning process or acquaintance with the task, boosting the performance to high accuracy in three sessions executed in days 6 and 7 of experiments with half-day separation (Fig. 5B and movie S2). By examining the confusion matrix in Fig. 5B, most of the errors were observed to be caused by the misjudgment of the medium stiffness toward either soft or hard. The soft object was never confused with the hard one and vice versa.

Furthermore, we tested whether the participant, who was blindfolded and in acoustic isolation, was able to use the differential recruitment of the restored sensation in different hand sites, to recognize object shape, keeping stiffness constant (hard/high, in this case). Three different items were independently presented (Fig. 5C): a cylindrical object (a bottle), a large spherical object (a baseball)—both of which covered the entire hand palm—and a small spherical object (a mandarin orange), which covered only part of the hand palm. The participant was able to correctly classify all three shapes with an average accuracy of 88% (Fig. 5D). Although the baseball was large enough to cover the entire hand, the subject could feel that the spherical shape produced a different sensation than the cylindrical bottle. To achieve this discrimination ability, the participant referred to the use of the perceived delay between the contact of the index and little fingers with the object surface. This delay was indeed significantly different between the spherical and the cylindrical shapes (Kruskal-Wallis, P < 0.01) (Fig. 5D and fig. S3).

DISCUSSION

In the case study presented here, sensory feedback was achieved by stimulating peripheral nerve fascicles, which, in turn, allowed real-time closed-loop control of a prosthetic hand by a human subject with a transradial amputation. To restore the lost sensory feedback, four TIMEs were implanted in the median and ulnar nerve fascicles, and two stimulation sites that were able to elicit distinct and repeatable sensations on the innervation territories of the two nerves (35) were then selected at the end of systematic testing of all the contacts and then connected to the artificial hand sensors. Sensations were elicited in a range from slight contact to just below the reported pain threshold, to dynamically control the intensity of stimulation delivered, according to the prosthetic hand sensor readouts.

The participant controlled the prosthesis through voluntary contractions of the remaining muscles on the stump, being able to perform different (ulnar, pinch, and palmar) grasps, and hand opening by online processing of sEMG signals. The grasps were performed in terms of position control such that he was able to finely modulate the fastening and opening of the prosthetic hand. The complex and fine manipulation tasks in the natural human hand require a deep and complex interaction between motor commands and sensory feedback (22, 23). Current solutions are not able to provide a natural (that is, physiologically appropriate) sensory feedback to amputees for real-time closed-loop prosthetic use. In the present study, we developed a system able to perform control and to deliver sensory feedback with an imperceptible delay (24) for the user. Our approach allowed the user to perform several tasks with very promising results. In the fine force control task, the subject indeed mastered a precise closed-loop control of the force elicited by his voluntary contraction, using either median or ulnar natural feedback of pressure intensity. With a series of “placebo” trials, it was confirmed that the high performance obtained was only due to the restored sensation, which allowed the user to master force control in the absence of any other feedback (visual or auditory). The possibility of adding physiologically appropriate touch sensations to hand prosthesis could enhance the controllability and, thus, acceptability (6, 7) of such a device, bringing it closer to the natural manipulation strategy.

This level of precise force control is not reachable with currently available prostheses, owing to the lack of natural sensory feedback offered to the user. Moreover, the participant’s performance rapidly increased during a week of tests and training, indicating that the participant intuitively and precisely integrated the information provided by the restored feedback in his control loop. This finding was also confirmed by the trials in which the participant recognized, through the sensory feedback, having exceeded the pressure level requested and corrected the grasping force accordingly. Moreover, the present approach featured an emerging, higher level of complexity than controlling only different levels of force of single fingers, when the participant was able to successfully integrate both the median and ulnar sensory feedbacks into a full-palm control of exerted pressure.

Successively, the participant’s ability to exploit the prosthetic sensory-motor loop in a task involving different object sensing and various available motor commands was tested. Our hypothesis was that this complex sensory integration could be exploited to gain control of the hand prosthesis in manipulation tasks, even without visual or auditory feedback. The participant rapidly mastered the task, demonstrating the ability of integrating the sensory information in real-time control for several grasps of the prosthetic hand. Thus, the subject achieved appropriate grasping and manipulation of some common objects. Therefore, single sensations dispatched along separate neural pathways can be combined to achieve a comprehensive, physiological, and functional prosthetic agency experience.

Restoring the sensory pathway should serve as a method not only to improve the controllability of the prosthesis but also to help in regaining the ability to explore the environment. Our hypothesis was that the close to natural sensory feedback delivered from the prosthesis sensors would allow the user to actively sense the environment, being able to recognize object properties, such as stiffness, size, and shape. Tests conducted with a baseball, a mandarin orange, and other visually dissimilar objects indicated that the subject was indeed able to intuitively integrate the sensory information received to recognize that the objects were different and therefore needed to be handled with differently grasps and levels of force. The subject probably used a discrimination approach very similar to the physiological one based on the tactile information provided by the different average slope of the force-time and contact-time curves associated with the fingertips for different objects (25). Stiffer objects reach the maximum of the perceived force faster than less stiff objects. Thus, the participant benefited from the same characteristic behavior in the sensation elicited by the nerve stimulation and encoded it to achieve discrimination of three different object compliances after only three sessions. The patient recognized the object stiffness in less than 3 s, which we believe is compatible with real-life applications (26).

The sensory information naturally received while exploring an object’s shape is a complex mixture of redundant and overlapping inputs coming from different sensory receptors and fibers (27, 28). However, as shown in our case, even sensing a differential recruitment of two parts of the hand is sufficient to recognize the shape of some familiar objects. For objects characterized by different shapes covering all the palmar prosthetic hand, the timing of activation of the two sensors is in fact different. The subject to discriminate among the objects can use this kind of information. The restored sensation can thus induce an artificial, albeit close to natural, neural coding that allows the subject to intuitively integrate the combined stimulation of different neural pathways without any training. The successful combined use on multiple channels provides clear evidence of the buildup of natural perceptions because of the homology and real-time properties of this neural coding. The participant’s ability to control different levels of grasping force, execute functional manipulations, and identify some simple object properties as three levels of compliance and three different shapes provides powerful evidence of the impact that this approach could have in real-life applications.

However, this study was conducted on one participant over a limited amount of time, and future studies will show on larger populations of amputees accurately the performance and limits of this artificially induced sensory feedback integration into the control of prosthesis. The other limitation is the fact that the tests were conducted continuously over the course of 1 week, so it is not clear whether the user would retain or even improve performances over a longer period of not being used. Moreover, many other sensations, or more sophisticated perceptions that might be elicited with this implants, were not tested.

Restoring sensory feedback is necessary to improve the usability of a hand prosthesis in daily life activities, where regaining control of the force output or being able to recognize object properties would increase the quality of life of people who suffer from hand amputation. The concept of this closed-loop bidirectional control, using a stimulating neural interface, could also be extended to enable the stimulation of a larger number of sites on the nerve implants. By coupling these locations for stimulation with the readouts of as many sensors embedded in the hand prosthesis, a wider variety of sensations could be delivered to the user, in terms of both position (for example, palm sensing) and type of sensation (for example, proprioception). To translate this technology to common clinical practice and even everyday use, several goals have to be achieved. First, the equipment used for stimulation should be miniaturized and fully implantable. The control unit for decoding of motor intention from sEMG signals and encoding of sensation by stimulation should be programmed on-chip and introduced in the socket of the prosthetic hand. Overall, this approach opens up new possibilities for hand prosthesis users, paving the way for the development of natural, dexterous, and effective bidirectional control of these devices.

MATERIALS AND METHODS

Study design

To verify whether a restored natural sensory feedback could be integrated in the user control loop of a dexterous prosthetic hand, we implanted TIMEs into the median and ulnar nerves of an amputee’s residuum. Touch sensations were elicited on the median and ulnar innervation territories and exploited in the bidirectional control of a sensorized prosthetic hand. In this case study, the experiments were aimed at testing the subject’s ability to modulate the grasping strength by measuring the force output with a pressure sensor, and to explore the possibility to integrate the sensations into functional tasks and for the identification of daily object physical properties. Because there are no references for this type of experiment, we aimed to make as high as possible number of trials (n > 700) within the limited time on disposition. More in particular, we made many trials in experiments with the force control (both for evaluation and for falsification) because that is the essential for proof of concept and also the basic principle for all other experiments done.

Data were acquired in several sessions distributed in 7 days. Sessions lasted as long as the subject was comfortable with the time spent. Therefore, trials were interrupted when the subject asked it. Data were considered outliers when they exceeded 2 SDs from the mean.

Subject recruitment

All procedures were approved by the Institutional Ethics Committees of Policlinic A. Gemelli at Catholic University, where the surgery was performed, IRCCS San Raffaele Pisana (Rome), where the experiments took place, and Campus Bio-Medico University, whose clinical personnel collaborated during the experiments. The protocol was also approved by the Italian Ministry of Health. One male participant (D.A.S.), age 36 years, was selected for the experiments from a group of 31 candidates with hand amputation because of the stump characteristics (transradial amputation and sufficient number of remnant muscles) and his psychophysical abilities (expert user of EMG-driven hand prostheses). He suffered a transradial left arm amputation 10 years ago, as a consequence of a traumatic event.

Bidirectional prosthesis and real-time control

The surgical procedure for implanting TIMEs is described in the Supplementary Materials and Methods. The bidirectional prosthesis comprised a set of commercial devices (Prensilia IH2 Azzurra robotic hand, 2 GRASS QP511 analogical amplifiers, Multichannel System STG4008 stimulator) and the TIMEs developed in the homonymous EU project. The artificial hand was connected to the stump of the volunteer by a custom-made socket (Ortopedia Italia). The artificial hand and the stimulator were controlled by custom-developed software in LabVIEW (National Instruments). The prosthetic hand was equipped with tension sensors measuring the force exerted by the index and the little fingers.

The users’ residuum sEMG signals were used to decode the intended grasp. Decoded hand motion was driven in terms of progressive position control, resulting in a gradual opening or closing of the hand. The sensors embedded in the hand were used as inputs for the delivery of the afferent neural stimulation. Current-controlled stimulation was delivered through the TIME active sites (1 in the median nerve and 1 in the ulnar nerve, with overall 56 active stimulating and 8 ground sites), eliciting a sensory perception reliably localized within the territories of the stimulated median or ulnar sensory fascicles. The stimulation was provided at fixed frequency and width of a biphasic train of pulses, whereas the current amplitude was modulated proportionally to the sensor readouts.

sEMG-based control

The sEMG signals were collected differentially from the five muscular positions. Data were acquired at 12 kHz, band pass–filtered (100 to 1000 Hz), and grouped into intervals of 100 ms. For each interval, features were extracted and fed to a multilayer perceptron (MLP) network, providing the classification output every 100 ms. Signal processing, feature extraction and validation, and decoding are described in detail in the Supplementary Materials and Methods.

Restoration of sensory feedback

The ability of the electrodes to elicit sensations by means of electric current stimulation was tested during different trials. The experiments consisted of stimulating single contacts of the four electrodes, with a train of cathodic rectangular biphasic pulses. The frequency of the delivered pulses was 50 Hz, and the length of the stimulation train was 500 ms for every trial. The injected charge was varied within the safety limits indicated for the electrodes by the manufacturers and by the ethical committee. Elicited sensations reported by the subject (type, location, and strength in a scale from 1 to 10) were recorded.

Two stimulation sites able to elicit sensations on the sensory innervation territories of the two nerves were selected: a touch sensation on index and thumb for the median innervated territory, and a touch sensation on the little finger for the ulnar innervated territory were reported by the subject (Fig. 1, D and E). Sensations were elicited in a range that went from slight contact to pain threshold (corresponding to 14 to 24 nC and 4 to 8 nC for median and ulnar nerves, respectively). These properties were exploited to dynamically control the intensity of stimulation delivered, accordingly to the prosthetic hand sensor readouts.

Transformation of sensor readouts in stimulation patterns (encoding)

The readout of the sensors embedded in the prosthetic hand was used as an input for a proportional delivery of afferent neural stimulation. Sensors positioned in the index and little fingers were used to acquire the level of contact applied on the two sides of the robotic hand. A real-time algorithm dedicated to the sensory loop was able to read both hand sensor outputs and to encode the respective sensory stimulation. Simultaneously, the algorithm dedicated to the control loop was able to acquire, process, and decode the sEMG signals and to deliver the motion command to the robotic hand. The details of the algorithm and sensor encoding are described in the Supplementary Materials and Methods.

Experimental design

Several protocols were designed and implemented to demonstrate that the restored sensory feedback could allow the participant to effectively use the bidirectional hand prosthesis. In particular, four experiments were carried out: fine force control task (n = 560 trials), functional exploration and grasping tasks (n = 52 repetitions), stiffness recognition (n = 66 repetitions), and object shape recognition (n = 32 repetitions). The details of each of these four experiments are in Supplementary Materials and Methods.

During all the experiments, the participant was blindfolded and acoustically shielded to eliminate both visual and auditory feedback. He did not receive any systematic and prolonged training, but he quickly learnt by himself how to use and control the bidirectional robotic hand.

Data collection and normalization

During the experiments with the bidirectional prosthesis, the following information were recorded: index and little finger sensor readouts, stimulation parameters (current amplitude, frequency, and pulse width), desired finger position, pressure sensor output, residual muscle stump sEMGs, and MLP decoded intention.

All the data traces shown in the paper are normalized for formatting and simplicity of visualization reasons. All the traces corresponding to the pressure sensor chamber were normalized with respect to the maximum pressure exerted by the hand, which was of about 1.5 kPa (after sensor calibration). Because hand sensors were measuring the tension in the fingers’ tendons, these values correspond to a measure of the force exerted at the tip of the finger, which was comprised in the range 0 to 60 N. Current values were saturated within the 15% (S15) to 75% (S75) of the sensor readouts (Fig. 1). Stimulation current traces were normalized to the maximum stimulation current: 240 μA (at 100 μs) for the index finger and 160 μA (at 50 μs) for the little finger.

Data analysis and statistics

The acquired data were exported and processed offline in MATLAB R2012 (The MathWorks). All data were reported as mean values ± SD or SEM when indicated. Performances were evaluated in terms of confusion matrices measuring the number of outcomes for each possible answer with respect to each requested task. Because data distributions were not Gaussian [Kolmogorov-Smirnov, 95% confidence interval (CI)], statistical evaluations were performed using the two-tailed Kruskal-Wallis test (a nonparametric analysis of variance) with 95% CI. The two-tailed Tukey-Kramer test was applied in the case of multiple groups of data.

SUPPLEMENTARY MATERIALS

www.sciencetranslationalmedicine.org/cgi/content/full/6/222/222ra19/DC1

Materials and Methods

Fig. S1. Force reproducibility.

Fig. S2. PCA analysis of the staircase task.

Fig. S3. Variability of sensor data in stiffness and shape recognition tasks.

Movie S1. Functional exploration and object handling.

Movie S2. Object stiffness recognition.

References (29, 30)

REFERENCES AND NOTES

  1. Acknowledgments: We are grateful to participant D.A.S. who freely donated 6 weeks of his life for the advancement of knowledge and for a better future of people with hand amputations. Funding: This work was supported by the EU Grant CP-FP-INFSO 224012 (TIME project), the project NEMESIS (NEurocontrolled MEchatronic hand prostheSIS) funded by the Italian Ministry of Health, the EU Grant FP7-611687 NEBIAS (NEurocontrolled BIdirectional Artificial upper limb and hand prosthesiS), and the EU Grant FP7-NMP 228844 NANOBIOTOUCH (Nano-resolved multi-scan investigations of human tactile sensations and tissue engineered nanobiosensors). The work was also supported by the Swiss National Science Foundation through the National Centre of Competence in Research Robotics. Author contributions: S.R. designed the study, developed the software, performed the experiments, and wrote the paper; M. Capogrosso, M.B., and F.M.P. developed the software and the overall system integration, performed the experiments, analyzed the data, and wrote the paper; J.R. developed the software and collaborated during the experiments; G.D.P. selected the patient and collaborated during the design of the study and during the experiments; J.C., G.G., L.C., A.L.C., and C.M.O. collaborated during the integration of all the components of the device and collaborated during the experiments; M. Controzzi, C.C., and M.C.C. developed the robotic hand; W.J. and E.G. collaborated during the preparation and execution of the experiments; T.B. and T.S. developed the TIME electrodes; E.F. performed the surgery; P.M.R. selected the patient, designed the study, supervised the experiments, and wrote the paper; and S.M. designed the study, supervised the experiments, and wrote the paper. All the authors read and approved the manuscript. Competing interests: C.C. and M. Controzzi have financial interests in the company Prensilia SRL, which manufactures artificial hands similar to the one used in this study; S.R., M. Capogrosso, M.B., F.M.P., and S.M. are inventors of a pending patent application concerning part of the methods in this work. Data and materials availability: Raw data and the custom-developed LabVIEW software will be made available by materials transfer agreement upon reasonable request to S.M.
View Abstract

Navigate This Article