Abstract
The automatic emotion recognition from facial expressions has become an exceptional tool in research involving human subjects and has made it possible to obtain objective measurements of the emotional state of research subjects. Different software and commercial solutions are offered to perform this task. However, the adaptation to cultural context and the recognition of complex expressions and/or emotions are two of the main challenges faced by these solutions. Here, we describe the construction and validation of a set of facial expression images suitable for training a recognition system. Our datasets consist of images of people with no experience in acting who were recorded with a webcam as they performed a computer-assisted task in a room with a light background and overhead illumination. The six basic emotions and mockery were included and a combination of OpenCV, Dlib and Scikit-learn Python libraries were used to develop a support vector machine classifier. The code is available at GitHub and the images will be provided upon request. Since transcultural facial expressions to evaluate complex emotions and open-source solutions were used in this study, we strongly believe that our dataset will be useful in different research contexts.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
The landmark datasets of the images are available at https://github.com/julian-tejada/EmotionRecognition
Code availability
The scripts used to classify the face expressions coming from a video are also available at https://github.com/julian-tejada/EmotionRecognition
References
Abdulrahman, M., & Eleyan, A. (2015). Facial expression recognition using support vector machines. In: 2015 23nd signal processing and communications applications conference (SIU) (pp. 276–279). https://doi.org/10.1109/SIU.2015.7129813.
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest. https://doi.org/10.1177/1529100619832930
Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., & Ton, V. (1997). Matsumoto and Ekman’s Japanese and Caucasian facial expressions of emotion (JACFEE): Reliability data and cross-national differences. Journal of Nonverbal Behavior, 21(1), 3–21. https://doi.org/10.1023/A:1024902500935
Boyko, N., Basystiuk, O., & Shakhovska, N. (2018). Performance evaluation and comparison of software for face recognition, based on Dlib and Opencv library. In: 2018 IEEE second international conference on data stream mining processing (DSMP) (pp. 478–482). https://doi.org/10.1109/DSMP.2018.8478556.
Calvo, M. G., Fernández-Martı́n, A., & Nummenmaa, L. (2014). Facial expression recognition in peripheral versus central vision: Role of the eyes and the mouth. Psychological Research Psychologische Forschung, 78(2), 180–195. https://doi.org/10.1007/s00426-013-0492-x
Card, S. K. (2018). The psychology of human–computer interaction. CRC Press. (google-Books-ID: iUtaDwAAQBAJ).
Casaletto, K. B., & Heaton, R. K. (2017). Neuropsychological assessment: Past and future. Journal of the International Neuropsychological Society: JINS, 23(9–10), 778–790. https://doi.org/10.1017/S1355617717001060
Chen, C., & Jack, R. E. (2017). Discovering cultural differences (and similarities) in facial expressions of emotion. Current Opinion in Psychology, 17, 61–66. https://doi.org/10.1016/j.copsyc.2017.06.010
Darwin, C., & Ekman, P. (2009). The expression of the emotions in man and animals, anniversary edition (4th ed.). Oxford University Press.
Durán, J., Reisenzein, R., & Fernández-Dols, J. M. (2017). Coherence between emotions and facial expressions. In J. -M. Fernandez-Dols, J. A. Russell (Eds.) The science of facial expression (1st ed., pp. 107–139). Oxford University Press.
Dzedzickis, A., Kaklauskas, A., & Bucinskas, V. (2020). Human emotion recognition: Review of sensors and methods. Sensors (basel, Switzerland). https://doi.org/10.3390/s20030592
Ekman, P., & Friesen, W. V. (1978a). Facial action coding system: Manual. Consulting Psychologists Press. (oCLC: 5851545).
Ekman, P., & Friesen, W. V. (1978b). Manual for the facial action coding system. Consulting Psychologists Press.
Ekman, P., & Heider, K. G. (1988). The universality of a contempt expression: A replication. Motivation and Emotion, 12(3), 303–308. https://doi.org/10.1007/BF00993116
Ekman, P., Rolls, E. T., Perrett, D. I., & Ellis, H. D. (1992). Facial expressions of emotion: An old controversy and new findings [and discussion]. Philosophical Transactions: Biological Sciences, 335(1273), 63–69.
Ekman, R. (1997). What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS). Oxford University Press.
Fleiss, J. L., Nee, J. C., & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974–977. https://doi.org/10.1037/0033-2909.86.5.974
Goeleven, E., De Raedt, R., Leyman, L., & Verschuere, B. (2008). The Karolinska directed emotional faces: A validation study. Cognition and Emotion, 22(6), 1094–1118. https://doi.org/10.1080/02699930701626582
Izard, C. E., & Haynes, O. M. (1988). On the form and universality of the contempt expression: A challenge to Ekman and Friesen’s claim of discovery. Motivation and Emotion, 12(1), 1–16. https://doi.org/10.1007/BF00992469
Jack, R., Garrod, O. B., & Schyns, P. (2014). Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current Biology, 24(2), 187–192. https://doi.org/10.1016/j.cub.2013.11.064
Jack, R. E., Sun, W., Garrod, O. G. B., & Schyns, P. G. (2016). Four not six: Revealing culturally common facial expressions of emotion. Journal of Experimental Psychology: General, 145(6), 708–730. https://doi.org/10.1037/xge0000162
Kortli, Y., Jridi, M., Falou, A. A., & Atri, M. (2018). A novel face detection approach using local binary pattern histogram and support vector machine. In: 2018 international conference on advanced systems and electric technologies (IC ASET) (pp. 28–33). https://doi.org/10.1109/ASET.2018.8379829.
Kortli, Y., Jridi, M., Falou, A. A., & Atri, M. (2020). Face recognition systems: A survey. Sensors (basel, Switzerland). https://doi.org/10.3390/s20020342
Kret, M. E. (2015). Emotional expressions beyond facial muscle actions. A call for studying autonomic signals and their impact on social perception. Frontiers in Psychology, 6, 711. https://doi.org/10.3389/fpsyg.2015.00711
Krumhuber, E. G., Küster, D., Namba, S., & Skora, L. (2020). Human and machine validation of 14 databases of dynamic facial expressions. Behavior Research Methods. https://doi.org/10.3758/s13428-020-01443-y
Lang, P. J. (1995). The emotion probe. Studies of motivation and attention. The American Psychologist, 50(5), 372–385. https://doi.org/10.1037//0003-066x.50.5.372
Lange, K., Kühn, S., & Filevich, E. (2014). “Just Another Tool for Online Studies” (JATOS): An easy solution for setup and management of web servers supporting online studies. PLoS ONE, 10(6), e0130834.
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I. (2010). The Extended CohnKanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition—workshops (pp. 94–101). https://doi.org/10.1109/CVPRW.2010.5543262(iSSN: 2160-7516).
Lundqvist, D., Flykt, A., Öhman, A. (1998). The Karolinska directed emotional faces—KDEF. [CD ROM] from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet
Mäkäräinen, M., Kätsyri, J., Takala, T. (2018). Perception of basic emotion blends from facial expressions of virtual characters: Pure, mixed, or complex? Computer Science Research Notes 2802. https://doi.org/10.24132/csrn.2018.2802.17.
Mathôt, S., Schreij, D., & Theeuwes, J. (2011). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314–324. https://doi.org/10.3758/s13428-011-0168-7
Matsumoto, D. (1992). More evidence for the universality of a contempt expression. Motivation and Emotion, 16(4), 363–368. https://doi.org/10.1007/BF00992972
Mehrabian, A. (1997). Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression. Journal of Psychopathology and Behavioral Assessment, 19(4), 331–357. https://doi.org/10.1007/BF02229025
Muller, E., Bednar, J. A., Diesmann, M., Gewaltig, M. O., Hines, M., & Davison, A. P. (2015). Python in neuroscience. Frontiers in Neuroinformatics. https://doi.org/10.3389/fninf.2015.00011
O’Toole, A. J., An, X., Dunlop, J., Natu, V., & Phillips, P. J. (2012). Comparing face recognition algorithms to humans on challenging tasks. ACM Transactions on Applied Perception, 9(4), 16:1-16:13. https://doi.org/10.1145/2355598.2355599
Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. Cambridge University Press. https://doi.org/10.1017/CBO9780511571299
Perez, F., Granger, B. E., & Hunter, J. D. (2010). Python: An ecosystem for scientific computing. Computing in Science & Engineering, 13(2), 13–21.
Plutchik, R. (2001). The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 89(4), 344–350.
Russell, J. A. (1991). The contempt expression and the relativity thesis. Motivation and Emotion, 15(2), 149–168. https://doi.org/10.1007/BF00995675
Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C. H., Xiang, Y., & He, J. (2019). A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors (basel, Switzerland). https://doi.org/10.3390/s19081863
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors (basel, Switzerland). https://doi.org/10.3390/s18072074
Skiendziel, T., Rösch, A. G., & Schultheiss, O. C. (2019). Assessing the convergent validity between the automated emotion recognition software Noldus FaceReader 7 and Facial Action Coding System Scoring. PLoS ONE, 14(10), e0223905. https://doi.org/10.1371/journal.pone.0223905
Stöckli, S., Schulte Mecklenbeck, M., Borer, S., & Samson, A. C. (2018). Facial expression analysis with AFFDEX and FACET: A validation study. Behavior Research Methods, 50(4), 1446–1460. https://doi.org/10.3758/s13428-017-0996-1
Takagi, S. (2015). Comparison of multisensory display rules in expressing complex emotions between cultures (pp. 57–62). http://www.isca-speech.org/archive/avsp15/av15057.html. Accessed 8 Aug 2019
Tipples, J. (2019). Recognising and reacting to angry and happy facial expressions: A diffusion model analysis. Psychological Research Psychologische Forschung, 83(1), 37–47. https://doi.org/10.1007/s00426-018-1092-6
Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249. https://doi.org/10.1016/j.psychres.2008.05.006
Tracy, J. L., Robins, R. W., & Schriber, R. A. (2009). Development of a FACS-verified set of basic and self-conscious emotion expressions. Emotion (washington, DC), 9(4), 554–559. https://doi.org/10.1037/a0015766
VandenBos, G. R. (ed.). (2007). APA dictionary of psychology (pp. xvi, 1024). American Psychological Association.
Young, A., Perrett, D. I., Calder, A., Sprengelmeyer, R. H., Ekman, P. (2002). Facial expressions of emotion: Stimuli and test (FEEST). Type: Other
Yu, L., Zhou, K., & Huang, Y. (2014). A comparative study on support vector machines classifiers for emotional speech recognition. Immune Computation, 2(1), 35–42.
Funding
JT was funding by the Fundación Universitaria Konrad Lorenz during his stay as visitor professor in that institution.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tejada, J., Freitag, R.M.K., Pinheiro, B.F.M. et al. Building and validation of a set of facial expression images to detect emotions: a transcultural study. Psychological Research 86, 1996–2006 (2022). https://doi.org/10.1007/s00426-021-01605-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00426-021-01605-3