1,000 years of emoji history and what Machine Learning means for its future

In one form or another, images have accompanied text for thousands of years. Providing greater depth and imbuing written content with enriched meaning, emoji have long served to evoke emotional responses and delight the reader. From the idiosyncratic and bizarre to the mundane, emoji have transformed over time in how and where they are used. This transformation is not done yet, however, as Machine Learning will bring about great change, as we are already seeing today.
The history of emoji and its future with Machine Learning

People have been enriching written text with pictures as long as writing has existed. In other words, emoji have been around a very long time. This was not necessarily because the picture served a specific purpose, but just for the fun of it, and to make the reader smile. Take the marginalia that medieval monks put in the margins of their manuscripts (hence the academic Latin name) for example. As you can see in the image below, these depictions were often light-hearted, comical depictions.

The history of emoji and its future with Machine Learning

Fast forward to the early 1990s. The WWW was still in its infancy, but chat services like Internet Relay Chat (IRC) were already hugely popular. In my previous life in Academia I did a study on the use of language in IRC, which was interesting because chat services combine some elements of verbal communication (e.g. the synchronicity) with the use of written language in a way no other medium had ever done before (Nils Lenke & Peter Schmitz: “Geschwätz im ‘Globalen Dorf’ – Kommunikation im Internet”, in: Osnabrücker Beiträge zur Sprachtheorie 50 (1995), pp. 117-141). But written language lacks some elements of spoken language, like paralinguistic features that signal irony or other “emotional” components of communication. Because these are not visible in plain, written words, people used a lot of emoticons like J or L to make up for this. So, these pictorial elements were not just fun to look at, they actually served an important purpose in communication.

Now in 2017, today’s emoji are not just in the margins of texts, but are everywhere – in chats, text messages, emails, tweets, posts, etc. And they combine the two elements we have previously discussed: on the one hand they are silly and fun to look at like the medieval marginalia– somebody even compiled a highly entertaining list of marginalia that look like emojis. On the other hand they serve an important function in conveying emotion and tone like emoticons. But there are many, many different emoji to choose from and today’s world is moving fast – searching from menus can just be too slow.

And now, modern keyboards like uggest emoji automatically. Even better, the keyboard has just been updated to proactively offer emoji suggestions with the help of Machine Learning. Previously this feature had been simply keyword based, which we felt wasn’t good enough. Often you need an emoji that captures the meaning of a whole phrase or sentence, so a keyword might not be good enough to make an appropriate suggestion.

Contemplating this challenge, my colleague Claes-Fredrik Mannby found that Machine Learning was the solution we needed, as is often the case these days. Rather than tediously mapping specific words to emoji by hand, he simply fed the Machine Learning algorithms lots and lots of examples of how people combine words and emoji. In the end, it all works quite nicely. If you want to see an example, just type or Swype something like “winter vacation” or “let’s hit the slopes” and the top choice emoji will be a pair of skis – and so on. Prepare yourself for smarter, contextually relevant emoji that will enrich communications everywhere with enjoyable, meaningful and easy to read text.  This capability is now in Swype for Android (on Google Play) and available as part of the Swype SDK, which will hit devices soon.


Get Swype on Google Play

Swype Keyboard is now even more customizable with new themes and Machine Learning-powered emoji

Get the app

Tags: , ,

Nils Lenke

About Nils Lenke

Nils joined Nuance in 2003, after holding various roles for Philips Speech Processing for nearly a decade. Nils oversees the coordination of various research initiatives and activities across many of Nuance’s business units. He also organizes Nuance’s internal research conferences and coordinates Nuance’s ties to Academia and other research partners, most notably IBM. Nils attended the Universities of Bonn, Koblenz, Duisburg and Hagen, where he earned an M.A. in Communication Research, a Diploma in Computer Science, a Ph.D. in Computational Linguistics, and an M.Sc. in Environmental Sciences. Nils can speak six languages, including his mother tongue German, and a little Russian and Mandarin. In his spare time, Nils enjoys hiking and hunting in archives for documents that shed some light on the history of science in the early modern period.