Photo by Aerps.com on Unsplash
LLMs, short for Large Language Models have become a common thing right now. And yea, some people still call it ‘AI’, but LLMs are only one use case of AI
Anyway, let me get back to my story…
So, LLMs have become so common at this point in time, It’s quite difficult to find someone who have not used an LLM. Even if you are totally against AI companies, there’s a great chance that you may have accidentally triggered an AI response on Google search.
If you are a developer, writer, scientist or even a teacher.. then you are probably wondering whether AIs gonna replace you.
But today’s article is not going to be about the ethical or social aspects of AI. I’m gonna talk about how LLMs work, internally.
If you have used any Agentic coding tool, or perhaps used an LLM through an API, you may have come across terms like, Parameters, Tokens, Embeddings, Context, Caching, Temperature etc. And I have used these LLMs for some time without having no idea how they work. That’s why I decided to break it down, and explain everything.
Some yapping about Machine Learning
If you have been reading this article carefully, you may have noticed me saying that “LLMs are only one use case of AI”. AI is a vast field of study, and inside that we have a field called Machine Learning. Which simply means teaching computers to do stuff.
Unlike in typical computing, where we come up with an algorithm that turns input into output, in machine learning we give the computer the input and output and ask it to come up with a statistical model to predict the closest output that an input can produce.
So for machine learning, and of course for building LLMs, we need a thing called a Neural Network.
And believe it or not, neural networks are inspired by our own human brain.
Core of it all: The humble neuron
A biological neuron
The above thing is called a neuron (yes the actual thing we see in human brains).
Neuron works like this. It has multiple inputs, and one output. Inputs are called dendrites and when a chemical input signal arrives at a neuron, it compares all the inputs and decides whether it should send an input or not.
So, similarly in computer science a digital neuron (essentially a mathematical function) takes some inputs and produces a corresponding output.
Neural Network: The actual brain
Neural network is.. well, a connected set of neurons. But they are connected in a special way. There are different layers, and each and every neuron’s output in one layer connects with the inputs of each and every neuron in the next layer.
In other words, if there are three inputs, then each neuron in the second layer must have three inputs, and if we have 5 such neurons, then each neuron in third layer must have 5 inputs.
If we look at the above example (shown in the picture) from a mathematical perspective. each neuron in second (blue) layer is a mathematical function with 3 variables, let’s say x,y,z
output = ax + by + cz + d
so, what are these a, b, c values? These values are called weights of our neural network. and d is called the bias. Altogether, these coefficients are called the Parameters of our neural network.
Phew, that’s one term cleared out.
Anyway, in our current neural network, we can create the blue neurons by setting different values for these a, b, c, d constants. And often, we don’t choose these values ourselves. These values are determined by a process called Training.
One other thing you need to know is that we can write each layer transition as a matrix multiplication
- X are input
- W areweights of blue layer
- B arebias of blue layer
- H being outputs of blue layer
Similarly, for the last transition
- V are the weights of green layer
- C are the bias of green layer
- Y are the two outputs of the neural network
I know, this is getting kind of boring, but trust me you need to know these matrix stuff to understand the rest.
Now, let’s see how this neural network can be used to build an LLM
Turning words into Math
Language models are for processing languages. But computers only understand math. That’s why we need a way to convert words into math.
So what we do first is convert words into numbers. We call these numbers as Tokens. A token can either be a letter, a part of a word, or even a single word.
For the sake of this article, let’s take a simple sentence “cat sat on the mat.” We can assign each word a unique number like cat = 1, sat = 2, on = 3, the = 4, mat = 5, . = 6
So we can represent the whole sentence as a tokenized array = [1,2,3,4,5,6]
Tokens are just numbers, and they themselves are not that useful. We need to turn each token into an Embedding. Simply what we need to do is, we send each token through a matrix called Embedding Matrix.
And what we get is a high dimentional vector (called Embedding vector). And this vector contains some key information that our token alone didn’t have:
- Embeddings represent the conceptual meaning of text mathematically. Words with similar meanings produce vectors that are closer together in vector space.
In modern LLMs this vector is usually between 4,096 and 12,288 numbers long for just one token
Back to Neural Networks
Now it’s time to feed our embedded tokens into the neural network.
As I said earlier, each token produces a vector of size bigger than 4096. And our actual prompt usually contains several hundred of these tokens.
We combine all these vectors into a matrix.
Input Matrix size = Number of tokens in prompt * Embedding dimension size
For a brief 100-token prompt in a standard model, the network’s first layer receives a grid of 100 columns by 4,096 rows.
And what do we get as output? We get a single token
Wait.. That makes no sense
Yes you heard right, if our prompt is “cat sat on a”, then we get an input matrix of size 4 * (Embedding size)
We pass it through our massive neural network, and our powerful GPUs compute the matrix values all in parallel.
This causes us two issues:
- Where are the rest of the output tokens?
- Didn’t we just discard the sequential ordering of our tokens by putting them in a matrix?
Let me answer them one by one.
Converting each token into an embedding vector and putting it in a matrix will definitely discard any positional value of that token. So what we can do instead is putting the positional value in the embedding itself.
Which means, two tokens, although having the same value, will produce two different embedding values depending on their position.
And for the other question,
The loop
To produce continous meaningful output from an LLM, we need to run it in a loop.
Loop looks like this. If we feed three words into an LLM “The cat sat”, then it will create three tokens (assume), and for each token we genrate an embedding vector (this time with position data)
so now we have a 3xE matrix. and we give it as input to our neural network, and hopefully it gives out the word “on”
Now we repeat. take the newly generated word, and append it to our input.
“The cat sat on” this creates a new input matrix of size 4xE
We put that matrix back in, and get another output, let’s say “the”
“The cat sat on the”
Put that new 5xE matrix back, and we get… “mat”
“The cat sat on the mat”
Now putting that 6xE matrix back we get… EOL
yes, there is a special tokens that LLMs can spit out to stop the loop. (otherwise it will get stuck in a miserable loop)
Have you ever noticed, when chatting with an LLM, you getting one word after another? That’s exactly what’s going on.
What about those Chat Bots
The chatbots we use are also built using these LLMs, but they have a some differences.
If you’ve been reading the article carefully, you may have already figured out the issue with traditional LLMs
Yes.. They suck at following instructions.
Fundamentally, and LLMs are just big ass predictors (the same kind you find in your smart keyboard). They just do a lot of statistical math and predict the most likely next word, considering the previous input.
But for this kind of an LLM if we ask a simple question like: “How do I find a girlfriend?” it will probably give the output like “I’m a 22 year old guy and I have never been…”
Well, the LLM doesn’t understand that you are asking a question and it will just make your question longer, instead of giving you an answer.
How to fix this?
To fix this, we need to train our LLM to understand questions, and follow instructions. Simply, what we do is just feed it thousands of example questions along with sample answers, untill the LLM figures out how to answer a question.
This kind of work usually require a lot of manual work, where people write questions and answers to feed to the LLM, and review it’s output.
Getting the Conversational Feel
Answering a question is just the beginning. We usually want that conversational feel, where we ask clarifying questions and expect the LLM to answer the new question also considering what was asked before.
Suprisingly, the way we do this is simple. We format the conversation in a way like this:
When a user asks a question like “Can you explain why sky is blue?”
We create a prompt like this:
<|system|> You are a helpful, rude AI assistant. Answer user's questions
<|user|> Hi, can you explain why the sky is blue?
<|assistant|>we separate each input with special tokens to define system prompt, user’s messages and assistant’s messages
And whenever user ask another question, like “Does the same thing happen in mars?”
We prepare an entirely new prompt, with the new message appended to the bottom
<|system|> You are a helpful, rude AI assistant. Answer user's questions
<|user|> Hi, can you explain why the sky is blue?
<|assistant|> The sky is blue because of a phenomenon called Rayleigh scattering...
<|user|> Does the same thing happen on Mars?
<|assistant|>and the LLM output will be like:
“No idiot, the mars has little to no atmosphere”
Getting LLMs to think
LLMs are a type of sequential statistical machine. They can predict the next word of a paragraph.
With these limitations, how can we get an LLM to give reliable, and meaningful output. And how can we get it to solve complex problems?
We introduce a thing called thinking..
Think about how you think.
Is it any different from talking? or writing?
No right?
We just think about stuff using the good old languages that we use to talk. We just talk with ourselves without making any sound.
That’s exactly what we need to do to allow LLMs to think.
We just add another type of token called
And when the LLM is confident that it came up with a good solution to your problem, it can output a token like
Understanding limits of AI
Most people think of LLMs as super intelligent computers that know everything.
That’s partially correct. But you need to understand the limits of LLMs and how you structure your prompts will reflect the quality of the output that the LLM is generating.
For example If you upload a math question and ask the LLM to give you only the answer then you will most likely get a wrong answer (assuming thinking mode is off)
Why? because If the LLM has never seen the question before, then there’s no statistical probability for it to produce the correct answer.
But if you give the same question, but change your prompt to something like this. “Solve this question step-by-step, showing all your work before delivering the final answer.” you will most likely get a better answer.
Limits of Computation
Some of the limits of LLMs come from the way LLMs are built, and development (or lack thereof) in computer science.
LLMs whole knowledge is stored in a bunch of numbers called Parameters. And since these are real numbers represented on a computer, these numbers have a real world limit to their bit size. And creating LLMs with larger bit depth parameters will not be feasible and scalable with current technology.
Also, one other aspect LLMs lack compared to humans is the memory. Today LLMs are trained on a specific dataset, and the trained model parameters are shipped with the model.
And these parameter values stay constant during the model’s execution. Preventing the model from gaining new knowledge, and forgetting old useless knowledge (like humans tend to do)
Although, it’s theoretically possible to build an ever evolving, continuously learning AI, it would not be feasible with today’s business models. (Imagine an LLM remembering things you said and telling it to some random guy in the internet)
Conclusion
Yea, that’s a lot of stuff to put in an article. If you enjoyed reading the article, let me know in the comments below.