Speech Feature Extraction for Emotion Recognition Using Machine Learning

dos Santos, A. N. Faculdade de Engenharia Elétrica e de Computação, Universidade Estadual de Campinas (Unicamp), Campinas, SP, Brasil, a264372@dac.unicamp.br

dos Reis, V. A. Faculdade de Engenharia Elétrica e de Computação, Universidade Estadual de Campinas (Unicamp), Campinas, SP, Brasil, v261503@g.unicamp.br

Masiero, B. S. Faculdade de Engenharia Elétrica e de Computação, Universidade Estadual de Campinas (Unicamp), Campinas, SP, Brasil, masiero@unicamp.br

Abstract

In machine learning (ML) supervised classification problems, it is often beneficial to crunch down the input data, mapping it to an initial pre-processing stage, to improve the performance of pattern recognition systems, such as artificial neural networks (ANNs). But the success of this kind of approach relies on a viable choice of features to best represent the essence of the input data. In this scenario, most ML sound applications make use of acoustic models, but, regarding machine hearing, this works better if some basic properties of the human hearing are considered, such as the variable widths of cochlear critical bands with respect to frequency. In this work, some techniques of cepstral feature extraction are investigated, based on linear predictive coding (LPC), which can model the source-filter behavior of glottal speech, and short-time Fourier transform (STFT), using triangular and gammatone shaped filter banks to warp the short-time power spectrum into different scales of frequency based on human auditory perception. In addition to it, a dimensional reduction of these representations is further accomplished by applying a discrete cosine transform (DCT). Subsequently, these feature extraction techniques are applied to a speech-only portion of an emotion recognition data-set, serving as a front-end to an ML model. This back-end consists of a shallow and fully connected architecture, known as linear classifier, which has just one input layer and one output layer, and is regularized via the least squares (LS) method. Finally, each model obtained is compared for accuracy. In our validation experiments, the gammatone cepstral coefficients (GTCC) was the most prominent front-end, with 55% overall accuracy. However, in our test experiments, the Mel frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LPCC) and Bark frequency cepstral coefficients (BFCC) features resulted in improved accuracy, with 98%, 94% and 94%, respectively.

Keywords:

cepstral features

speech emotion recognition

linear predictive coding

artificial neural networks

regularized least-squares.

References

[1] Simon Haykin. Neural networks and learning machines, 3/E. Pearson Education India, 2009.

[2] Eric J Humphrey, Juan P Bello, and Yann LeCun. Feature learning and deep architectures: New directions for music informatics. Journal of Intelligent Information Systems, 41(3):461–481, 2013.

[3] V Kishore Ayyadevara. Pro machine learning algorithms. Berkeley: Apress, 2018.

[4] Jakob Abeßer. A review of deep learning based methods for acoustic scene classification. Applied Sciences, 10(6), 2020.

[5] Richard F Lyon. Human and machine hearing. Cambridge University Press, 2017.

[6] Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018.

[7] The open speech repository. URL https://bit.ly/3zh JVOH.

[8] K Sreenivasa Rao, V Ramu Reddy, and Sudhamay Maity. Language identification using spectral and prosodic features. Springer, 2015.

[9] Haytham Fayek. Speech processing for machine learning: Filter banks, mel-frequency cepstral coefficients (mfccs) and what’s in-between. 2016. URL https://bit.ly/3wVFhEg.

[10] Juan I Godino-Llorente, Santiago Aguilera-Navarro, and Pedro Gómez-Vilda. Lpc, lpcc and mfcc parameterisation applied to the detection of voice impairments. In Sixth International Conference on Spoken Language Processing, 2000.

[11] Lawrence R Rabiner. Digital processing of speech signals. Pearson Education India, 1978.

[12] Behnam Fazlali and Mohammad Eshghi. A pipeline design for implementation of lpc feature extraction system based on levinson-durbin algorithm. In 2011 19th Iranian Conference on Electrical Engineering, pages 1–5. IEEE, 2011.

[13] Bruce P Bogert. The quefrency alanysis of time series for echoes; cepstrum, pseudo-autocovariance, cross-cepstrum and saphe cracking. Time series analysis, pages 209–243, 1963.

[14] Panos E Papamichalis. Practical approaches to speech coding. Prentice-Hall, Inc., 1987.

[15] Abel Herrera and Fernando Del Rio. Frequency bark cepstral coefficients extraction for speech analysis by synthesis. The Journal of the Acoustical Society of America, 128(4):2290–2290, 2010.

[16] Richard Bronson. Schaum’s Outline of Matrix Operations. McGraw Hill Professional, 1988.

[17] Gilbert Strang. Linear algebra and its applications. Belmont, CA: Thomson, Brooks/Cole, 2006.

[18] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.

[19] Mingke Xu, Fan Zhang, and Wei Zhang. Head fusion: improving the accuracy and robustness of speech emotion recognition on the iemocap and ravdess dataset. IEEE Access, 9:74539–74549, 2021.

[20] Soonil Kwon. Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network. International Journal of Intelligent Systems, 36(9):5116–5135, 2021.

[21] Tanvi Puri, Mukesh Soni, Gaurav Dhiman, Osamah Ibrahim Khalaf, Ihtiram Raza Khan, et al. Detection of emotion of speech for ravdess audio using hybrid convolution neural network. Journal of Healthcare Engineering, 2022, 2022.