Researchers from the Nizhny Novgorod branch of the Higher School of Economics have created a system that recognizes emotions in human speech. The neural network is just learning, but in the future such technology can significantly reduce the “gap in understanding” between a person and a computer.
Russian researchers work with artificially created intelligence – the neural network. It is a mathematical model, built on the principle of the networks of nerve cells of a living organism. To teach her to recognize emotions in human speech, scientists used audio recordings.
The sounds of the human voice turned into images similar to medical diagnostic cards or thermal imager data. These are three-dimensional diagrams, formed in two colors: green and red.
“The redder the color, the more energy, the more the volume was.” For example, we see that “evil” and “happy” are the most red, that is, they are the strongest emotions, hence there is more energy that we do not see, for example, in a neutral or calm emotion, or frightened, “says one of the developers, Alexander Rassadin.
The neural network was allowed to listen to 1000 audio files in which 24 people (English speakers) pronounced phrases with different emotional expression. “We had eight emotions: happiness, anger, neutral emotions, calm emotions, surprise, sadness, fright and disgust.” Our program recognizes them with an accuracy of 71%, “says the author of the work Anastasia Popova.
The sounds in the microphone create electrical vibrations that help mathematically transform the sound into an image – a chalk-spectrogram. According to the spectrogram of sound, musicians can draw definite conclusions about music, for this they do not need to listen to it. The same can be done by a program with emotions.
The neural network in the process of learning seeks patterns in images marked by the same emotions, forms an internal representation of each emotion. While the program successfully distinguishes between neutral and tranquil intonations, and worst of all recognizes happiness and surprise: happiness often confuses with fear and sadness, and surprise – with disgust. A separate task for scientists is to understand what kind of laws the neural network has learned.
“It’s probably impossible to achieve the accuracy of 100% recognition, rather, we want to train this program on more extended data so that it can be more competent to recognize any entry in any language, not necessarily in English, not necessarily this phrase – different phrases, and we do not want it to depend on the language, we do not care what language the person speaks in order to understand his psychological state, we want to achieve this from our program, “says Anastasia Popova.
While the researchers did not even think about commercializing their project. For them, this is just the beginning of a lot of work, which will make communication between a person and a computer more interactive. The point is that the technician will be able to understand his owner right up to his psychological state and take some measures if necessary.
“The creation of such an instrument closes a gap in communication between a computer and a person.If the computer learns to recognize human emotions, then all the applications that are currently available immediately become intellectually on the head.There are also some voice assistants, and now they are deprived of emotions, as soon as they are endowed with them, they can be much more effective, “says Alexander Rassadin.
Scientists suggest that the program for recognizing emotions in speech can be applied in medicine, and in the work of different call centers, in robotics. This system can solve problems in creating a “smart home”.
“I, as one of the developers of the smart home system, know that there is a serious enough problem to recognize some bad behavior that occurs in an apartment: thieves have climbed, someone with a knife began to attack – often this is indistinguishable from the fact that children are raging around the couch and the creation of such a tool can finally allow this step to become a real intellectuality, “says Alexander Rassadin.