Facial Displays and Their Dialogical Meanings



Language and Logic Courses

Advanced Course

Facial Displays and Their Dialogical Meanings,
Jonathan Ginzburg (Paris Diderot University, France) and Ye Tian (Amazon Research Cambridge, UK & Laboratoire de Linguistique Formelle, Université Paris Diderot, France)

Week 2, 9:00 – 10:30, Room 267 moved to Mirror Hall, Floor 2

The course aims at modelling the discourse impact of non-verbal social signals (nvss) such as laughter, smiling, frowning, and the like, but also including conventionalised pictorial representations (“emoji”) as textual manifestations. We will start with a brief overview of the two main theoretical sources for this course: first, work in computational and theoretical linguistics on the semantics and pragmatics of dialogue and second, psychological and computational approaches to emotion appraisal. The course will offer detailed argumentation, contrary to received wisdom until recently, as to the mutual interaction between nvss/emojis and content emanating from verbal material, in particular that nvss/emojis bear propositional content. We propose viewing nvss/emojis as akin to an event anaphors and show how to deduce various pragmatic functions from the basic meanings posited in combinination with enthymematic reasoning and emotional appraisal. The course will also include practical sessions on multidimensional laughter classification of speech using Praat and emoji data analysis in social media.

Course Outline

Lecture 1: Previous work on laughter and humour; motivation for a semantic/pragmatic approach; the KoS framework for dialogue. ginzburg-tian-esslli-2018-lek1


  1. Darwin, C. (1877). The expression of the emotions in man and animals. Oxford University Press, USA. With an introduction and notes by Paul Ekman.
  2. Morreall, J. (2016). Philosophy of humor. In Zalta, E. N. (Ed.), The Stanford Encyclopedia of Philosophy (Winter 2016 edition). Metaphysics Research Lab, Stanford University.
  3.  Attardo, S. (2008). A primer for the linguistics of humor. The primer of humor research, 8, 101–157.
  4. Spatharas, D. (2006). Persuasive ΓeΛΩΣ: public speaking and the use of laughter. Mnemosyne, 374–387.

  5. Jefferson, G. (2004). A note on laughter in ‘male–female’ interaction. Discourse Studies, 6(1), 117–133.

  6. Cooper, R. & Ginzburg, J. (2015). Type theory with records for NL semantics. In Fox, C. & Lappin, S. (Eds.), Handbook of Contemporary Semantic Theory, Second Edition, pp. 375–407 Oxford. Blackwell.

Lecture 2: the KoS dialogue framework (cont); A minimal semantic/pragmatic theory of laughter. http://esslli2018.folli.info/wp-content/uploads/ginzburg-tian-fsm-essli18-lek2-1.pdf


1. Ginzburg, J. (2016). The semantics of dialogue. In M. Aloni and P. Dekker (Eds.), The Cambridge Handbook of Formal Semantics. Cambridge: Cambridge University Press.

2. Breitholtz, E. & Cooper, R. (2011). Enthymemes as rhetorical resources. In Artstein, R., Core, M., DeVault, D., Georgila, K., Kaiser, E., & Stent, A. (Eds.), SemDial 2011 (Los Angelogue): Proceedings of the 15th Workshop on the Semantics and Pragmatics of Dialogue.

3. Ginzburg, J., Breitholtz, E., Cooper, R., Hough, J., & Tian, Y. (2015). Understanding laughter. In Proceedings of the 20th Amsterdam Colloquium University of Amsterdam.


Lecture 3: A minimal semantic/pragmatic theory of laughter. (Cont.)  Classifying laughter in corpora using Praat.

First part: http://esslli2018.folli.info/wp-content/uploads/fsm-essli18-lek3-1.pdf

Second part: http://esslli2018.folli.info/wp-content/uploads/tian-ginzburg-201808_ESSLLI_laugh.pdf


Lecture 4: Cognitive theories of emotion; Incorporating cognitive theories of emotion into KoS;  A formal analysis of emojis.

First part: http://esslli2018.folli.info/wp-content/uploads/fsm-esslli18-lek4.pdf

Second part:


  1. Scherer, K. R. (2009), ‘The dynamic architecture of emotion: Evidence for the component process model’, Cognition and emotion 23(7), 1307–1351.
  2. Gratch, J. & Marsella, S. (2004). Technical details of a domain-independent framework for modeling emotion. ICT tech report.

    3. Herring, S., Stein, D., & Virtanen, T. (Eds.). (2013). Pragmatics of computer-mediated communication (Vol. 9). Walter de Gruyter.

    4. Dresner, E., & Herring, S. C. (2010). Functions of the nonverbal in CMC: Emoticons and illocutionary forceCommunication theory20(3), 249-268.

    5. Novak, P. K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentiment of emojisPloS one10(12), e0144296.

    6. Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., & Hecht, B. (2016). Blissfully happy” or “ready to fight”: Varying Interpretations of EmojiProceedings of ICWSM2016.


  3. Lecture 5: Analyzing emojis in social media; A formal theory of non-verbal social signals in dialogue: laughter, smiling, frowning, sighing; Incorporating non-verbal social signals into the grammar.
  4. Hoey, E. M. (2014). Sighing in interaction: somatic, semiotic, and social. Research on Language and Social Interaction, 47(2), 175–200.

  5. Niedenthal, P. M., Mermillod, M., Maringer, M., & Hess, U. (2010). The simulation of smiles (sims) model: embodied simulation and the meaning of facial expression. Behavioral and brain sciences, 33(6), 417–433.

    4. Ginzburg, J. & Poesio, M. (2016). Grammar is a system that characterizes talk in interaction. Frontiers in Psychology, 7, 1938.