Spelling with Brain Signals

At a Glance:

Can a computer work out which letter someone wants?

The person using this spelling system watches letters and commands flash on a screen while focusing on the one they want. Sensors on the scalp record electrical activity from the brain. That recording is called EEG. The computer looks for differences between the response to the intended choice and the responses to everything else.

The signal of interest is the P300, a brief change in brain activity that can follow a relevant flash. It is not a direct recording of a thought or a letter. The system has to infer the choice from noisy measurements collected over repeated flashes.

For this one-month course project, I used recorded EEG data to train and evaluate a model, then checked whether its predictions recovered a hidden word. The broader idea is communication without a conventional keyboard. My work was an offline student prototype, not a live assistive device.

Turn the recording into something the model can compare

I used Python to divide each recording into short slices around a flash: 200 milliseconds before it and 800 milliseconds after it. These slices are called epochs. Lining them up this way lets the model compare responses at the same point after each flash.

I focused on the 200–600 millisecond portion from three sensor positions, Fz, Cz, and Pz, running along the middle of the scalp. For each position, the code calculated five averages of the signal. These 15 numbers are the features: a compact description of one trial instead of the entire recording.

I trained Linear Discriminant Analysis (LDA), a model that scores how closely a response resembles the examples labeled as targets. A target is a flash containing the intended command; a non-target is a flash without it. The saved run used 600 training trials and 600 trials from a separate validation recording to check performance.

Python · Train on one recording, check another

clf = LinearDiscriminantAnalysis(
    solver='lsqr', shrinkage='auto'
)
clf.fit(X_train, y_train)

auc = roc_auc_score(
    y_val,
    clf.decision_function(X_val)
)

Reading the code. fit learns from the training examples and their labels. decision_function scores the separate validation examples; roc_auc_score compares those scores with the known answers. The saved result was 0.7204. Source: PA4.ipynb, training and validation cells; imports and data preparation omitted.

A useful signal, but not a correct word

The validation score was AUC 0.7204. AUC checks how well the model ranks target responses above non-target responses. Here, it ranked a randomly chosen target above a randomly chosen non-target about 72% of the time. A score of 0.5 is chance-level ranking and 1.0 is perfect. This is not the percentage of letters spelled correctly.

To get from flash responses to letters, I grouped the validation trials into five blocks, one for each intended selection. I added the model’s scores for the commands flashed in each block and picked the command with the highest total. Then I compared those choices with the recorded target word.

Five intended letters, five predicted commands

Target and predicted command at each of the five positions
Position	Target	Prediction	Result
1	Q	B	Mismatch
2	U	Bs	Mismatch
3	A	A	Match
4	C	C	Match
5	K	K	Match

The hidden target was QUACK. The notebook printed BBsACK: B, Bs, A, C, K. “Bs” is one command label, so the output represents five selections.

Three letters right still leaves the wrong message

The hidden word was QUACK, and the output was BBsACK. The final three selections—A, C, and K—matched. The first two did not. “Bs” is a single command label, which is why five selections produced six printed characters.

That result is the most useful part of the project for me as a designer. The model always picked a winner, even when the winner was wrong. If the interface displayed that choice as settled text, the person using it would have to deal with the mistake after the fact.

Give the person a say in the prediction

The errors suggest a simple design rule: show the prediction as a suggestion before adding it to the message. A short ranked list could keep alternatives available, while a confirmation action would let the person accept or reject the choice. Undo belongs nearby, because a wrong selection should not mean rebuilding the message.

An interface idea based on the errors

SuggestShow the likely letters.
CheckLet the person accept or choose again.
AddKeep the last choice easy to undo.

A design concept, not a tested interface.

Explain what the system knows

Confidence feedback should be understandable too. A message such as “these choices are close” describes uncertainty without pretending the model knows the answer. The accumulated scores in this notebook are not verified probabilities that a letter is correct, so they do not support a precise “90% confident” label.

The same principle applies while the system is learning from the user’s responses, a process called calibration. A progress message can explain that it is still collecting examples. Otherwise, a person may be left wondering whether the system is working or waiting for them. These are design implications of the analysis, not measured improvements in communication.

What this project showed

The model found enough signal to identify three of five intended letters in one word. That is a limited result from one course dataset, not a general spelling-accuracy estimate. There was no usability study with representative users or medical validation, and this prototype is not a medical device.

There is also an implementation limit: the averaging windows in the notebook overlap because of the way their starting positions are calculated. The figures here are the saved notebook results with that behavior, not results from a corrected version.

The project made one thing clear: predicting a letter and helping someone communicate are different tasks. Getting three letters right matters, but so does giving the person a way to deal with the two that were wrong.