Research ArticleNEUROTECHNOLOGY

Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface

See allHide authors and affiliations

Science Translational Medicine  11 Nov 2015:
Vol. 7, Issue 313, pp. 313ra179
DOI: 10.1126/scitranslmed.aac7328
  • Fig. 1. Neural signal nonstationarities.

    (A) Session timeline. Following an open-loop reference and decoder initialization block, a standard decoder was calibrated using several closed-loop center-out blocks, each lasting 3 to 5 min. Using the standard decoder, the participant then typed words, phrases, sentences, and/or paragraphs in either a QWERTY or a radial virtual keyboard. An RTI-based decoder was calibrated using only the neural data acquired during typing, and the participant continued typing using an RTI decoder until the end of the session. (B) Mean threshold crossing rates in each block of an example session (participant T7’s trial day 293), showing each channel that was used by the decoder for at least one block in the session. Blocks are labeled as in (A). Every third channel is labeled with its electrode number (in this session, 80 of 192 possible channels were selected for decoding in each block). For better visualization of the dynamic range of rate changes across blocks, rates are capped at 50 Hz (the highest actual whole-block baseline rate in this session was 68.3 Hz). (C) Directional tuning of the same channels in (B), obtained by regressing firing rates against target directions. Color represents the estimated PD, and the brightness of the color is proportional to the channel’s normalized modulation index. (The same PDs are shown in polar coordinates in fig. S3.) (D) The difference between each unit’s baseline rate in each block (“actual”) and the rate used by the decoder in that block (“used”; that is, the previous block’s baseline rate) is plotted against the difference between that unit’s baseline rate in that block and its rate in the first block (“original”), which would have been used by the decoder for the whole session if features were not being updated. (E) The angular difference between each unit’s measured PD in each block and the tuning model used by the decoder for that unit in that block (“actual vs. used”) is plotted against the angular difference between the measured PD in that block and in the first block (“actual vs. original”), which would have been the tuning model if the decoder were never recalibrated after the first block.

  • Fig. 2. Bias correction.

    (A) Representative example of bias estimation, from 800 s into the first typing block of participant T7’s first self-paced typing session (trial day 293). At each moment in time, the direction and magnitude of the velocity bias (red arrow) were estimated by computing an exponentially weighted running mean of all decoded velocities (grayscale dots) whose speeds exceeded the 66th centile of the speed distribution (red dashed circle) computed from the previous filter build. This threshold was empirically found to be high enough to exclude low-velocity movements generated in an effort to counteract existing biases. The shading of each dot represents time, with darker dots occurring closer to the present moment [the end of the highlighted period in (C)]. (B) Effect of bias correction at the same moment displayed in (A). The location of the cursor is represented as a black dot. The location of the (retrospectively inferred) target is a blue dot. The red arrow represents the estimated bias at that moment in time [same as in (A)]. The purple arrow indicates the decoded cursor velocity at that moment before bias correction. The blue arrow indicates the bias-corrected velocity. (C) Effect of bias correction on this entire block of typing. (Top) Velocity traces with the estimated bias (black traces) added in. The gray box indicates the time interval when individual velocity samples are displayed in (A). (Bottom) Actual cursor velocities that occurred in session, bias correction having been continuously applied in real time.

  • Fig. 3. RTI decoder calibration.

    (A) To obtain a tuning model from data acquired during neural control in a practical BCI application, such as a virtual keyboard (14), the user’s intended movement direction is retrospectively inferred to be directly toward the next selected target (white arrows). The white curve reflects the portion of the preceding cursor trajectory assumed to result from the person’s movement intent and is used toward RTI decoder calibration. The red dashed segments of the trajectory are excluded from decoder calibration. The intended direction vectors are regressed against the corresponding neural activity to calibrate the RTI decoder. (B) Typing performance using the RTI decoder versus the standard decoder, measured using the number of CCPM. Data are from 19 sessions across four participants, including 5 self-paced typing sessions (3 from participant T6 and 2 from participant T7, shown in unfilled markers). The within-session correlation coefficient r and its corresponding P value are shown in plot. (C) Typing performance using the RTI decoder versus the standard decoder, measured using the number of CSPM. For the radial keyboard, this metric can be translated into extrapolated bitrate (eBR = CSPM × log2(N − 1)/60, where N = 8 targets). The eBR scale only applies to the radial keyboard sessions, not to the two sessions in which the QWERTY keyboard was used (*); for the QWERTY keyboard, eBR could not be computed easily because of the large variability in the size of the targets. Within-session correlation coefficient and P value are shown in plot. P values in (B) and (C) were obtained by comparing the measured value to a null distribution obtained by shuffling the pairings 1,000,000 times.

  • Fig. 4. Example of self-paced typing session for participant T6 on trial day 668.

    In the self-paced typing sessions, participants were able to pause typing when they wanted by selecting the right arrow in the radial keyboard and then the wedge containing “PAUSE.” Each pause initiated a file break and RTI decoder build, and then neural control was restored to allow the user to resume when desired, by selecting the right arrow and “UNPAUSE.” Until the unpause sequence was selected, no other wedges were active. (A) Photograph of the radial keyboard interface (left) with the PAUSE button about to be selected, and the notebook showing the text typed in this session (right). (B) Length of each block of typing, the number of CCPM and CSPM in that block, and the text entered (the vertical lines in the text of the last block indicate an “ENTER” character, which starts a new paragraph). In this session, an RTI decoder was calibrated during each of the self-timed pauses using all typing data acquired up to that point, except the last RTI decoder used only the previous three typing blocks. Note that the fastest typing rate in this session was achieved in the last typing block. The blurred words, represented by underscores in B, were redacted at the request of the participant.

  • Fig. 5. All self-paced typing sessions: Summary of typing rates over time.

    Each session is depicted in a single hue, with darker bars indicating the time and duration of the self-paced blocks of typing in which a standard decoder was used, and lighter bars indicating the blocks in which an RTI decoder was used. Self-paced blocks of typing using an RTI decoder inthat same session are depicted in bright colored bars of the same hue. (A) Three self-paced typing sessions for participant T6. (B) Two self-paced typing sessions for participant T7. (C) One session with T6 in which bias correction, feature tracking, and RTI decoder calibration were all turned off. Linear regression between time and CSPM: Pearson’s correlation coefficient r = −0.85, P < 0.001. (D) One session with T7 in which bias correction, feature tracking, and RTI decoder calibration were all turned off (black bars). Linear regression between time and CSPM: r = −0.87, P < 10−6. In this session, T7 was unable to type at all in the third block; this occurred early enough in the session to test whether neural control could be rescued by reinstating the self-calibration methods (brackets). In the first and third rescue blocks, both bias correction and interblock feature tracking were reinstated, but the standard decoder was used (dark green bars). In the second rescue block (light green bar), an RTI decoder was used that was calibrated using data from the first rescue block.

  • Fig. 6. Self-calibration across multiple sessions for participant T6.

    Data are in the same format as Fig. 5 (the dark bar indicates the block in which a standard decoder was used, and the light bars indicate blocks in which an RTI decoder was used). The dots above the bars and the diamonds below the bars indicate typing periods during which the cursor’s speed gain or click decoder threshold, respectively, were manually adjusted by the technician; in the last two sessions of this series, there was no technician intervention once typing started. Using the self-paced radial keyboard, participant T6 typed whatever she wished across six sessions spanning 42 days, pausing and unpausing the BCI whenever she wanted, without needing to perform any explicit calibration tasks after the first day. The first block of the first session in this series (participant T6’s trial day 759) used a standard decoder calibrated earlier that day; after that, an RTI decoder was calibrated during every user-timed pause in neural control using the data acquired during the previous 20 to 60 min of typing. Each session after the first was initialized with the previous session’s last RTI decoder.

Supplementary Materials

  • www.sciencetranslationalmedicine.org/cgi/content/full/7/313/313ra179/DC1

    Materials and Methods

    Fig. S1. Schematic of all possible changes in cosine tuning.

    Fig. S2. Nonstationarities in mean threshold crossing rates during and between blocks of neural control.

    Fig. S3. Directional tuning, example session.

    Fig. S4. Simulation showing that RTI calibration can accommodate known shifts in PDs.

    Fig. S5. Self-paced typing session, participant T7’s trial day 293.

    Fig. S6. Spike panels from participants T6 and T7 without versus with common-average referencing.

    Table S1. Summary of participants.

    Table S2. Sessions contributed by each participant for each experiment.

    Movie S1. Self-paced typing session, participant T6’s trial day 668.

    References (4953)

  • Supplementary Material for:

    Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface

    Beata Jarosiewicz,* Anish A. Sarma, Daniel Bacher, Nicolas Y. Masse, John D. Simeral, Brittany Sorice, Erin M. Oakley, Christine Blabe, Chethan Pandarinath, Vikash Gilja, Sydney S. Cash, Emad N. Eskandar, Gerhard Friehs, Jaimie M. Henderson, Krishna V. Shenoy, John P. Donoghue, Leigh R. Hochberg

    *Corresponding author. E-mail: beata{at}brown.edu

    Published 11 November 2015, Sci. Transl. Med. 7, 313ra179 (2015)
    DOI: 10.1126/scitranslmed.aac7328

    This PDF file includes:

    • Materials and Methods
    • Fig. S1. Schematic of all possible changes in cosine tuning.
    • Fig. S2. Nonstationarities in mean threshold crossing rates during and between blocks of neural control.
    • Fig. S3. Directional tuning, example session.
    • Fig. S4. Simulation showing that RTI calibration can accommodate known shifts in PDs.
    • Fig. S5. Self-paced typing session, participant T7’s trial day 293.
    • Fig. S6. Spike panels from participants T6 and T7 without versus with common-average referencing.
    • Table S1. Summary of participants.
    • Table S2. Sessions contributed by each participant for each experiment.
    • Legend for movie S1
    • References (4953)

    [Download PDF]

    Other Supplementary Material for this manuscript includes the following:

    • Movie S1 (.mov format). Self-paced typing session, participant T6’s trial day 668.

Stay Connected to Science Translational Medicine

Navigate This Article