Transformers: The New Gem of “Deep Learning”

In recent years, Transformers have experienced rapid progress in Natural Language Processing

(Original version of this post in catalan)

We spend most of our time communicating in natural language, that is, in unstructured text, whether written or spoken. For this reason, research work has been going on for many years to ensure that machines understand these unstructured texts and extract relevant information from them. It is what is known as Natural Language Processing (NLP), one of the many areas of knowledge in the field of Artificial Intelligence.

Until recently, it was a true challenge for any of the multiple approaches used in NLP to understand the nuances of natural language. But in 2017, the research paper ‘Attention is all you need’ appeared, proposing a new neural network architecture called Transformer. It was definitely a turning point that completely changed the field of NLP.

Without realizing it, we often interact with Transformers. For example, already in 2019 Google introduced Transformers to its search engine to improve results. An example that Google itself explains in this post is when someone typed “2019 brazil traveler to usa need a visa” into their search engine (Figure 1). As humans, it’s easy to understand that this is someone who wants to go from Brazil to the United States and needs a VISA.

Figure 1: Google search results before and after using Transformers (Google Image).

Here, the word “to” and its relationship to the other words in the query are critical to understanding the meaning. But the algorithms that Google applied earlier did not understand the importance of this connection and returned results on American citizens traveling to Brazil (left part of Figure 1). Transformers can capture this nuance and know that a word as common as “to” has a very important role here, and, in this case, allow the search engine to show a result much closer to the user’s purpose (right side of the figure).

All this has meant that, in recent years, Transformers in NLP have experienced rapid progress (Figure 2 shows a…