Understanding ChatGPT
ChatGPT is essentially a chatbot, a computer program that mimics human conversations. These chatbots rely on Natural Language Processing (NLP) to interpret human inputs and deliver relevant responses. NLP is an area of AI that focuses on how computers interact with human language.
ChatGPT, developed by OpenAI, uses the GPT (Generative Pre-trained Transformer), a large language model.
What are Large Language Models?
Large Language Models (LLMs) are AI models that learn and generate human-like language. Well-known examples are OpenAI's GPT, Google's LaMDA, and Meta's LLaMA. LLMs form a key part of NLP. They study a ton of text data and then use this knowledge to understand or produce new text.
In a nutshell, LLMs predict the next word in a series based on the previous words. For instance, given the sentence "Jack and Jill went up the", an LLM predicts the next word, completing it as "Jack and Jill went up the hill". They achieve this through a method called machine learning.
What is Machine Learning?
Machine learning involves algorithms that learn from large datasets to make decisions. Consider this analogy: if you teach a kid various shapes through a picture book, they'll start recognizing a square or a circle by recognizing recurring attributes. Machine learning operates similarly. To predict the next word in a sentence, a computer analyzes a vast amount of text data, learning patterns and rules of language. These learned patterns form a model that the computer uses to predict the next word.
The backbone of machine learning used in LLMs is the neural network.
What is a Neural Network?
A neural network is a type of machine learning that emulates the human brain. It consists of small components called nodes or neurons, arranged in layers that collectively learn from data.
The increase in data availability and advancements in organizing nodes led to an increase in the number of nodes and layers, reaching into millions and billions, creating large neural networks. This new breed of algorithms with extensive neural networks is referred to as deep learning.
In 2017, Google introduced a new type of neural network called the Transformer which greatly enhanced NLP.
What are Transformers?
Imagine negotiating a holiday destination with your family. You don't just understand their suggestions based on individual propositions, but you take into account the whole dialogue. Transformers work similarly, especially with language.
A transformer is a type of neural network exceptionally good at context. It doesn't just consider the word it's currently processing and its immediate predecessor. It examines all the words in a sentence, identifies the relevant ones to the current word, and uses them to better comprehend the current word.
This ability stems from the attention mechanism that transformers use, allowing them to pay attention to different parts of the input data based on relevance. This makes transformers suitable for tasks like machine translation and text generation, where understanding the complete context is vital.
Most recent LLMs, including GPT, are built on transformers.
What are Tokens?
LLMs actually work on tokens, not words. A token is a piece of text. Common short words usually correspond to one token, while long and less common words are divided into multiple tokens. You can see how text is divided into tokens using OpenAI's Tokenizer.
Choosing the size of tokens involves a trade-off. Using each character as a token keeps the total number of tokens small but carries less information. On the other hand, using each word as a token can handle common and made-up words efficiently but can struggle with new or domain-specific words.
Learning Methods: Supervised and Unsupervised
Understanding how ChatGPT works involves getting to grips with some fundamental concepts of machine learning. Let's start with supervised and unsupervised learning.
Supervised learning is like having a teacher. For instance, if we want a machine to sort news articles into categories like sports or business, we first give it articles that we've already categorized. The machine learns from these examples.
Unsupervised learning is like learning through experience. We give the machine lots of data, but we don't tell it what to do with it. The machine has to figure out how to organize the data itself.
Most models for processing natural language (words and sentences) were trained using supervised learning. They could only do specific tasks, like classifying text or analyzing sentiment. Also, they required large amounts of labeled data, which is hard to find.
ChatGPT is first trained to learn by itself (unsupervised learning) and then fine-tuned for specific tasks (supervised learning).
Customizing ChatGPT
Fine-tuning is a way to customize ChatGPT for a specific job, making it more helpful and efficient. We take the version of ChatGPT that has learned a lot from a large amount of text data, and then we further train it on a particular task. This means we don't have to build a whole new model each time, which saves time and resources.
Models can also be used without any fine-tuning. These versions are called foundation models.
Types of Models
Depending on what the model is trained for (or fine-tuned in the case of foundational models), they can perform the following:
- Completion model: You give it a sentence, and it completes the rest.
- Conversational model: It's trained on conversation data to understand and participate in a dialogue.
- Instruction model: It's trained to follow human instructions.
- Question-and-answer model: It's trained to answer questions accurately.
Using ChatGPT
While using ChatGPT, we have to be careful because sometimes it can provide incorrect information.
This can happen because ChatGPT doesn't understand things like humans do. It's based on patterns it learned from its training data. Also, it can be sensitive to how a question or instruction is worded.
But don't worry, there are ways to reduce these mistakes or 'hallucinations'. For instance, we can give it better context data and design our questions or prompts well.
Prompts: Guiding ChatGPT
A prompt is a message that we use to guide ChatGPT's responses. Writing a good prompt is important for getting accurate and useful responses from ChatGPT. Here are some tips for writing good prompts:
- Be clear and simple.
- Clearly separate different parts of the prompt.
- Ask ChatGPT to check for conditions before answering.
- Give examples of the response you want.
- Break the task into steps.
- Keep refining the prompt until you get the response you want.
Conclusion
ChatGPT is an exciting tool. Understanding how it works can help us use it more effectively and by using it wisely, we can make it a useful assistant in many different tasks.
References
What Is ChatGPT Doing… and Why Does It Work?
