The way a machine reads
Understanding how AI processes our commands.

Mankind has always tried to merge technology and humanity, to integrate humans’ adaptability with machines’ superiority. In November 2022, humanity made a step in that direction through ChatGPT.

ChatGPT is a chatbot developed by OpenAI. Just like any other chatbot, ChatGPT holds conversations with users, teaches users about any topic, and produces original writing. Unlike other chatbots, ChatGPT is flexible. It adapts to any scenario with ease and improves through human interaction. It’s almost like a human.

Machines cannot read English or any other language. The only language machines process is ones and zeroes. Computer scientists developed programming languages like C, Java and Python to communicate with machines.

So, how can ChatGPT understand the commands we give it? We aren’t using a fancy coding language, just regular words. How does a machine read our questions and understand us? Natural Language Processing (NLP) answers these questions.

What is NLP?

Natural language processing is a field of artificial intelligence (AI) that creates techniques computers use to analyze and represent human languages. The reason such a field exists is because of the complexities of the human language. Consider the phrase, “I saw the Golden Gate Bridge flying into San Francisco.”

A reader who has never heard of the Golden Gate Bridge, or even of bridges in general, may read the sentence as “the Golden Gate Bridge” flew into San Francisco. In this situation, the reader did not have the appropriate context to understand the sentence.

Machines have no context whatsoever. They lack awareness of the cultural nuances that accompany the human language. The human language is ambiguous. Every word has multiple synonyms, each with different connotations that can impact your understanding of the sentence. Many words, like “can” and “orange,” can have different meanings based on the context.

Despite these problems, scientists devised techniques to process language. Recently, the most popular language processing technique comes from deep learning. Deep learning is a subset of AI that uses neural networks.

What are neural networks?

Warren McCulloch and Walter Pitts, researchers at the University of Chicago, proposed the idea of neural networks in 1944. They created a new computing model, modelled on the human brain.

A network of neurons composes the human brain. Every thought and action we do originates from a sequence of activated neurons. From what we experience with our senses, a signal is sent to a neuron. This neuron then sends signals to another sequence of neurons, where each neuron slightly modifies the signal. 

A sequence of neurons stores information. When we access information, we activate the neurons related to it. When we learn something new, we make new neural connections within our brain to store that information. Humans can build new knowledge from prior knowledge based on this biological process. And McCulloch and Pitts suggested neural networks as a way for machines to learn and create new information like humans.

Similar to a human’s network of neurons, neural networks would have a network of nodes. Each node acts like a neuron: it takes in a signal, modifies the signal, and passes the modified signal to the next relevant neuron. When a machine receives input, like when you type something into ChatGPT, that input is passed into the neural networks. The input starts a chain of node activations. Each node receives, modifies, and passes on a signal to the next node. At the end of the chain of activations, the neural network transmits the output: ChatGPT’s response to your question.

Computer scientists took the idea McCulloch and Pitts suggested in 1944, and continually improved upon it. In the modern day, different types of neural networks exist for different tasks. One type of neural network used for natural language processing is a recurrent neural network. 

Recurrent Neural Networks

In 2013, Richard Socher, Christopher Manning and Andrew Y. Ng introduced recurrent neural networks (RNN). These neural networks rely on the sequential nature of reading: how you can only read one word at a time. 

An RNN processes text by processing each word separately. It collects information on each previous word and utilizes that info to predict what the sentence says. By the end of the sentence, an RNN can predict how the sentence will end and what it is about.

However, this technique is flawed. RNNs place a great deal of focus on reading each character one at a time. As the algorithm only reads one character at a time, it is relatively slow at processing input. Another drawback is that the RNN fails to accurately understand long sentences. The more complex a sentence is, the more information the RNN has to keep track of. Eventually, the RNN loses track of important information from the start of the sentence. 

Transformers, a different type of neural network, fixed these issues.

Transformers

In 2017, a group of Google researchers presented new research at the 31st International Conference on Neural Information Processing Systems. They proposed transformers—a new self-attention-based neural network to replace RNNs in decoding sentences in NLP. 

A self-attention mechanism, or attention mechanism for short, gives each component of a sentence an attention score. An attention score ranks the relevance of each word in understanding the sentence as a whole. By calculating the attention score for each word, transformers track the most important parts of the sentence that determine the ‘meaning’ of the sentence. The score helps machines better understand the user input. Furthermore, a transformer analyzes every word in the sentence at the same time, making it more efficient than an RNN.

Transformer-based AI had wide-reaching impacts on machine learning and natural language processing. Its comprehension skills and computational efficiency revolutionized AI. Effective, human-like chatbots like ChatGPT only exist thanks to transformer architecture.