AudioVisual Brain Computer Interface

AV BrainMuscle Communication Interface

EEG & BCI: Brain Computer Interface

Brainwaves or electroencephalograms (EEG) have been measured since the 1920s, and since around 1970 it has been possible to allow a person, or a monkey, to control a device by "thinking", often after extensive training or biofeedback.

Commonly known as BCI, a Brain Computer Interface is classically thought of as a Human Computer Interface or HCI, but often the device that is ultimately controlled is not so much a computer as a physical mechanism, and often a prosthetic device which is a form of assistive technology designed to replace a limb or capability that a patient lacks due to disease or injury.  This leads to the use of the related term neuroprosthetics, as well as a number of alternatives and expansions of BCI like BMI, Brain Communication Interface or Brain Control Interface, Brain Machine Interface, etc. Many common approaches to BCI make use of EEG, but other biomedical signals and technologies can also be used, as well as combination with other E*G (ECG/EOG/EMG) electrodes as well as unrelated biomedical sensors and imaging techniques.

The different technologies and signals can be characterized as much by the tradeoffs that they involve as by the technical details of the methodology.  We focus on EEG, but also deal with other kinds of signal as appropriate – and in some cases hybrids of multiple technologies are useful, whilst at times other biological signals interfere with or are mistaken for EEG.

Invasive vs Non-Invasive

One of the major advantages of EEG is that it does not involve inserting electrodes inside the head.  For example, electrcorticography (ECoG) or intracranial EEG (iEEG) actually involves craniotomy, cutting through the skull to place electrodes directly on the cortex, the outer layer of the brain. Most of our student subjects would prefer we didn't do this, so we don't. The technique is mainly used with patients where the crainotomy would need to be performed anyway, or where a patient is expected to derive considerable direct medical benefit from the procedure. In fact, ECoG has been also used in experiments on animals since the 1950s. However our BCI research works only with non-invasive surface EEG, and one of our research goals is to minimize the inconvenience and discomfort that is involved even with traditional EEG.

Hundreds and Thousands

Some of the competitors to EEG costs millions of dollars and are inherently immobile, so while such technologies can usefully be used for research, they are not practical for actual use as a neuroprosthetic device for an ambulatory patient, or to help a quardaplegic patient walk. Magnetic Resonance Imaging (MRI) is in this category, and we use it only to help get the geography right for a subject we will be working extensively with using EEG. Functional MRI (fMRI) can be used to get good localization of where things happen in the brain, and can also be used as the technological base for BCI targeted to specific applications or treatment. Similarly MEG equipment is expensive, and although it provides an alternate dimension to EEG, and can be used with EEG in a complementary way, we confine our focus to the much cheaper EEG technology, for which our medical grade laboratory equipment costs only hundreds of thousands, a new generation of portable medical/research grade equipment costs only tens of thousands, and now consumer or games level equipment costs only a couple of hundred dollars, whilst being as easy to don as an audio headset.

Space and Time

There are far more technologies and techniques available for Neuroimaging than we can review here, but each tends to have its own advantages and disadvantages. One of the advantages of EEG is that it tends to be amongst the best techniques in terms of temporal resolution – routinely getting down to the order of a millisecond, and with modern technology even greater resolution is possible. However EEG electrodes are spaced several centimetres apart, on the surface of the scalp, and the resolution achievable, and the depth achievable, are in general better with other technology.  However, one of our research projects is focussed on changing this – with EEG electrodes located only millimetres apart.

Brain vs Muscle

Sometimes signal that doesn't actually originate as neural activity in the brain is misidentified as EEG.  In fact our unique research has shown that much of what has been identified in the past as brain signal actually isn't - particular in frequency ranges above about 20Hz (notably the so-called Gamma band) as well as in electrode locations nearer the periphery of the scalp (near muscle sources that are so strong that they even contaminate the lower frequency bands). In general these contaminants are all muscle of some form or an others, but many specific forms of contamination or "artefact" (or "artifact") are labelled for the particular organs whose muscle signal is detect.  Generally we call the muscle signal EMG, but the signal from the eye muscle is called EOG (Ocular) and the signal from the heart is called ECG or EKG (Cardiac in either English or German). Traditionally, experimental trials or other samples that are recognizably contaminated by artefacts due to eyeblinks and the like are simply deleted in their entirety.  More recently we are able to decompose or localize the components of the signal to some degree, and eliminate such identifiable muscle signal without having to throw the baby our with the bathwater.  This is a maor focus of our EEG research.

Input vs Output
Actual vs Imagined
Natural vs Artificial
Control vs Monitored
Thought vs Perception
Conscious vs Unconsicous

 

The original idea of BCI was that we could control a device by thinking, but the older idea of EEG is that we could monitor cognitive function. This is as radical a difference as the difference between input and output. But there is another extension beyond the idea of controling a device by thinking. We can also pick up information that relates more directly to perception, as well as information that relates to learning, or affective/emotional state. Many approaches to BCI involve trying to control "alpha" or "beta" – that is characteristic frequencies that occur in particular brain states.  Other approaches to BCI make use of imagination - imagining doing or perceiving something. These imagined movements might be used to make a cursor, or a wheelchair, move right or left.  But for a person with no ability to move themselves, the distinction between attempted, natural real movement and these artificial imagined movement blurs. These techniques often depend on biofeedback, training the person to think what the computer wants as much as the computer learning to recognize what the person is thinking or intending. Yet another approach to BCI makes use of basic perceptual and intentional correlates, and uses only the natural signals that occur as we perceive and react to external stimuli.

Multimodal Interfaces

There are many EEG-based techniques we can use for BCI, and our projects are exploring this entire space, and beyond.  Some of the techniques depend on providing an appropriate perceptual signal, such as flashes at particular frequencies and locations.  Some of the techniques related to intended or imagined movement. Some of them relate to internal states that accompany actual speech, gestures or expression. Why should we limit ourselves to EEG when their are other real world signals available that can provide additional information?  At heart our research is multimodal – research in BCI connects to research in other areas of HCI in a natural way, and information derived or transmmitted from any of these technologies can be combined, or fused.

AudioVisual BrainMuscle Communication Interface

KIT has research foci in Audio and Speech Processing, in Visual, Graphical and Haptic interfaces, in Robotics, and in Medical Technologies beyond EEG and BCI. Rather than treating muscle, EMG, as the enemy, or relying on it blindly as if it were EEG, or treating each modality as having to provide a solution on its own, we are seeking to knowingly combine together the information derived from different biosignal sources with those derived from our electronic technologies. The applications we are exploring range from interfaces to games, information retrieval and office products, assistive technologies and educational technologies.

Current Projects

  • ABC Wheelchair – using audio/speech, visual/IR/sonar and EEG/BCI to make use of all available AudioVisual, Brain/Muscle and Computational/Electronic technologies and modalities to achieve reliable and purposeful control of a wheelchair.

  • BCI-enabled Games – not just replacing conventional inputs with BCI input, but giving another dimension to the game by innovative augmentation of player capabilities.

  • Customization of ABC technology to individual people and their specific needs and disabilities – including those who are tetraplegic (quadraplegic) or locked in.Exploration of ABC technology as providing novel mechanisms for utilizing modern computer and communication technology for everyday users as well as those with disabilities.

  • Development of newer higher resolution EEG technology designed to revolutionize BCI and enable tracking of the formation the synchronization effects that mediate the formation of complex concepts: "synchrony and binding".

  • Development of unique methods for understanding the nature of the muscular contamination of EEG, and optimizing the separation and localization of the different signals.

Relevant Courses and Topics

The following awards and topics link directly to research within the AI Lab and final year projects are supervised for the courses shown (for further degree combinations see the individual course descriptions):

Bachelor of Computer Science (Honours) 
Bachelor of Behavioural Science (Psychology)/Bachelor of Computer Science
Bachelor of Engineering (Biomedical)
Bachelor of Engineering (Electronics)
Bachelor of Engineering (Robotics)
Bachelor of Engineering (Software)  
Bachelor of Information Technology (Honours) 
Bachelor of Information Technology (Digital Media) (Honours) 
Bachelor of Science (Honours) – Cognitive Science or Computer Science
Master of Engineering (Biomedical) 
Master of Engineering (Electronics) 
 

COMP3742 Intelligent Systems

COMP3751 Interactive Computer Systems
COMP3752 Computer Game Development
COMP4712 Embodied Conversational Agents
COMP4715 Computational Intelligence
COMP4716 Information Retrieval and Text Processing
COMP4717 Mobile Application Development 
ENGR3721 Signal Processing
ENGR3741 Physiological Measurement
ENGR3771 Robotic Systems
ENGR4722 Haptic Enabled Systems
ENGR4761 Image Processing
 

Recent Grants

ARC Discovery Grant  No. DP0988686 — Powers, Clark, Pope, Willoughby: Heterodensity EEG (2009-12)
"Heterodensity neuroimaging techniques for spatiotemporal identification and localization" - $301K

ARC Discovery Grant No. DP110101473 — Pope, Willoughby, Powers, Lewis (2011-13)
“Enhanced brain and muscle signal separation verified by electrical scalp recordings from paralysed awake humans" - $225K

ARC Linkage and Equipment Funding Grant No. LE0989734 (2010-12)
"Australian Speech Science Infrastructure: An Audio-Video Speech Corpus of Australian English"
 

Recent Publications

Refereed journal articles
Fitzgibbon, S.P., DeLosAngeles, D., Lewis, T.W., Powers, D.M.W., Grummett, T.S., Whitham, E.M., Ward, L.M., Willoughby, J.O. & Pope, K.J., Automatic determination of EMG-contaminated components and validation of independent component analysis using EEG during pharmacologic paralysis, 2016, Clinical Neurophysiology 127(3):1781-1793
  
Atyabi, A., Luerssen, M.H., Fitzgibbon, S.P.,  Lewis, T.W. &  Powers, D.M.W., Reducing training requirements for brain-computer interfaces through evolutionary dimension reduction and subject transfer, 2015, Neurocomputing.
 
Duan, LJ., Xu, YH., Yang, Z., Ma, W. & Powers, D.M.W., Transition Detection and Sample Purification for EEG Based Brain Computer Interface Classification, 2015, Journal of Medical Imaging and Health Informatics 5(4):871-875
 
Fitzgibbon, S.P., DeLosAngeles, D., Lewis, T.W., Powers, D.M.W., Whitham, E.M., Willoughby, J.O. & Pope, K.J., Surface Laplacian of scalp electrical signals and independent component analysis resolve EMG contamination of electroencephalogram, 2015, International Journal of Psychophysiology 97(3):277-284
 

Atyabi, A., Luerssen. M.H. & Powers, D.M.W., PSO-Based Dimension Reduction of EEG Recordings: Implications for Subject Transfer in BCI, 2013, Neurocomputing.

Fitzgibbon, S.P., Lewis, T.W., Powers, D.M.W., Whitham, E.M., Willoughby, J.O., & Pope, K., 2012. Surface Laplacian of Central Scalp Electrical Signals is Insensitive to Muscle Contamination. IEEE Transactions on Biomedical Engineering.

Whitham, E.M., Fitzgibbon, S.P., Lewis, T.W., Pope, K., DeLosAngeles, D., Clark, C.R., Lillie, P., Hardy, A., Gandevia, S.G., & Willoughby, J.O., 2011. Visual experiences during paralysis. Frontiers in Human Neuroscience, 5(160).

Pope, K., Fitzgibbon, S.P., Lewis, T.W., Whitham, E.M., & Willoughby, J.O., 2009. Relation of gamma oscillations in scalp recordings to muscular activity. Brain Topography, 22(1), 13-17.

Whitham, E.M., Lewis, T.W., Pope, K., Fitzgibbon, S.P., Clark, C.R., Loveless, S.J., DeLosAngeles, D., Wallace, A.K., Broberg, I.M., & Willoughby, J.O., 2008. Thinking activates EMG in scalp electrical recordings. Clinical Neurophysiology, 119, 1166-1175.

Whitham, E.M., Pope, K., Fitzgibbon, S.P., Lewis, T.W., Clark, C.R., Loveless, S.J., Broberg, I.M., Wallace, A.K., DeLosAngeles, D., Lillie, P., et al., 2007. Scalp electrical recording during paralysis: Quantitative evidence that EEG frequencies above 20 Hz are contaminated by EMG. Clinical Neurophysiology, 118(8), 1877-1888.

Fitzgibbon, S.P., Powers, D.M.W., Pope, K., & Clark, C.R., 2007. Removal of EEG Noise and Artifact Using Blind Source Separation. Journal of Clinical Neurophysiology, 24(3), 232-243.

Fitzgibbon, S.P., Pope, K., Mackenzie, L., Clark, C.R., & Willoughby, J.O., 2004. Cognitive tasks augment gamma EEG power. Clinical Neurophysiology, 115(8), 1802-1809.

Moores, K.A., Clark, C.R., Hadfield, J.L., Brown, G., Taylor, J., Fitzgibbon, S.P., Lewis, A., Weber, D.L., & Greenblatt, R., 2003. Investigating the generators of the scalp recorded visuo-verbal P300 using cortically constrained source localization. Human Brain Mapping, 18(1), 53-77.

Refereed conference papers

Cottrell, J., Fitzgibbon, S.P., Lewis, T.W., & Powers, D.M.W., 2012. Investigating a Gaze-Tracking Brain Computer Interface Concept Using Steady State Visually Evoked Potentials. Spring World Congress on Engineering and Technology.

Atyabi, A., Luerssen, M.H., Fitzgibbon, S.P., & Powers, D.M.W, 2012. The Impact of PSO based Dimension Reduction on EEG classification. Brain Informatics.

Atyabi, A., Luerssen, M.H., Fitzgibbon, S.P., & Powers, D.M.W., 2012. Dimension Reduction in EEG Data using Particle Swarm Optimization. IEEE Congress on Evolutionary Computation (CEC).

Atyabi, A., Luerssen, M.H., Fitzgibbon, S.P., & Powers, D.M.W., 2012. Adapting Subject-Independent Task-Specific EEG Feature Masks using PSO. IEEE Congress on Evolutionary Computation (CEC).

Atyabi, A., Fitzgibbon, S.P., & Powers, D.M.W., 2012. Multiplication of EEG samples through Replicating, Biasing, and Overlapping. Brain Informatics.

Atyabi, A., Fitzgibbon, S.P., & Powers, D.M.W, 2012. Biasing the Overlapping and Non-Overlapping Sub-Windows of EEG recording. International Joint Conference on Neural Networks (IJCNN).

Atyabi, A., Luerssen, M.H., Fitzgibbon, S.P., & Powers, D.M.W., 2012. Evolutionary feature selection and electrode reduction for EEG classification. IEEE Congress on Evolutionary Computation (CEC).

Atyabi, A., Fitzgibbon, S.P., & Powers, D.M.W., 2011. Multiplying the Mileage of Your Dataset with Subwindowing. Brain Informatics, Lecture Notes in Computer Sciences, Vol 6889, 173-184.

Yazdani, N., Khazab, F., Fitzgibbon, S.P., Luerssen, M.H., Powers, D.M.W., & Clark, C.R., 2010. Towards a Brain-Controlled Wheelchair Prototype. Proceedings of the 24th BCS International Conference on Human-Computer Interaction.

People