Neurolinguistic BCIs

Affiliated researchers: PD Dr. Tonio Ball, Dr. Johanna Derix, Olga Glanz

Most neurolinguistic and brain-computer-interface (BCI) studies to date have been conducted in experimental settings. Accordingly, little is currently known about neuronal activity during real-life, experimentally unconstrained natural communication. Since BCI technologies for speech restoration target real-life conditions of application, investigation of neuronal activity underlying natural speech is required.

Our studies (Derix et al. 2012, 2014; Ruescher et al. 2013) have shown that it is indeed possible to take neurolinguistic research to such conditions by evaluating extraoperative electrocorticography (ECoG) recordings obtained for pre-neurosurgical diagnostics in epilepsy patients.

We have extended this non-experimental approach by building up a multimodal corpus of authentic, uninstructed and spontaneous speech data with aligned ECoG recordings (Fig. 1). To this end, we used simultaneously recorded audio, video, and neuronal data.

Figure 1: Components of the multimodal speech-ECoG corpus. (a) The multimodal corpus consists of simultaneous video/audio and ECoG data. (b) The patients' speech is transcribed with PRAAT. 

This corpus provides a unique opportunity to perform linguistically grounded encoding studies on the neuronal basis of real-life speech production in the human cerebral cortex. Furthermore, it allows refinement of speech decoding approaches to the peculiarities of real-world linguistic and neuronal data, which can aid development of speech-BCI devices.

Our interdisciplinary research group at the interface of neurorobotics, machine learning, linguistics and clinical neuroscience aims to develop novel systems for human-to-robot interaction via speech-related brain signals. Our neurolinguistic research is grounded in non-experimental, spontaneous speech production, and it addresses multiple levels of linguistic abstraction (phonological, semantic, and syntactic). We are able to describe the linguistic material owing to methodological support from the Department of German linguistics at the University of Freiburg, which focuses on natural, usage-based approaches to studying language. Another on-the-spot cooperation partner is the Department of Computer Science, which provides expertise in the areas of robotics and cutting-edge machine learning methods.

The central innovation of this joint venue is to develop closed-loop devices for improved assisted communication. We seek to achieve this by taking speech decoding to real-world conditions and by using novel speech-based BCI systems optimally tailored to the conditions of real-world behavior (Fig. 2).

Figure 2: Concept of novel speech-based BCI systems (WISPER project, Baden-Württemberg Stiftung). Material modified from own previous publications: (1) and (2) from Derix et al. (2012), (3) from Thinnes-Elker et al. (2012).