Ah, hello everyone. (sighs) This is a test recording,
uh, for the talk I'm going to give in the Ooty Retreat, uh,
in June 2025. My name is Samuel Shadrack.
It's 3:00 PM right now, so the
talk will go till 5:00 PM, you know, assuming this test recording works out.
I'm going to introduce machine learning and introduce large language models at
a technical level. Uh, (door thuds) I'm going to be covering a lot of material,
so if you're completely new to this,
it's possible you will not (person coughs) understand all of it, but you'll understand
most of it.
The point is that you get an overview, not that you understand every single detail, you
know, just within two hours. That is going to require, you know, more homework exercises,
more practice, more, you know, writing your own code,
you know, deriving the formulas yourself. Like, that will take a bit more time.
It can't be done in two hours.
But once you have this overview, then you will be better placed to, you know, do more of
that learning even on your own or following in an online course.
I'm assuming a background of
basically, like, class 12,
you know, science background, so I'm assuming you know calculus, I'm assuming you know
differential equations.
Uh, not much of differential equation, but yeah, you should understand basic calculus,
you should understand matrix multiplication.
Uh, we are going to use concepts of calculus and matrices in today's
talk.
Okay, so let's begin. First, let's start with the problem.
So this is a very famous problem in machine learning called MNIST.
Uh,
MNIST stands for Modified NIST. So NIST is the standards body in the
US, and also many countries have their own st- standards bodies,
uh, that define, you know, various units,
like, you know, what is a meter, what's a kilometer? You know, how big should a map be?
What is one degree on a map? You know, these types of things.
So, they have defined a standard
dataset, which is for optical character recognition.
So basically, you have these photos of people who have written various letters...
not letters, sorry, numbers. So people have written, you know, three or two or seven or
one. Like, any number they've written on a piece of paper.
We have taken a photo of that piece of paper.
Uh,
we have converted it more specifically into a 28 by 28 pixel image.
It's grayscale, so it's not, you know, RGB. It's not red, green and blue.
It's just
either black or white or gray. So the gray value can be between zero and one.
So, you know, 0.8 value means, you know, that pixel is
very close to black, and maybe a 0.2 means it's very close to white.
Now we have, you know, 60,000 such photographs.
We already know which classes they're put into,
and now we want to write a program that will take an image as input and tell you, you
know,
what, uh, number is this. "Is this zero? Is this one?
Is this two?" And, like, we already know the answers, but
we don't want the program to memorize the answers.
Uh, you know, o- one way to solve this problem would be just, you know, we already know
the answers, why not just have like a dictionary in the (laughs) program that says, you
know, "If you notice the first image, say this. If you notice the second image, do this.
If else, if else, if else," you know, 60,000 times.
But we are not going to do that, and there's a reason why we are not going to do that.
The reason we are not going to do that is because we also want this program to work
on, uh, images which we have not seen.
So apart from these 60,000 photos, there's another 10,000 photos which we
have not seen. Like, we do not know what they are.
I mean, they're actually there on the internet, but right now we are not allowed to look
at them. That's the rule.
And we want our program to also perform well on these 10,000 images,
and those are also, again, you know, photos of zero, one, two, three, four.
So,
we can't hard code it. We need some actual, you know, way of solving this problem so that
it will work on the photos we have never seen.
I'm going to give everyone, like,
maybe two minutes. Just try thinking
how would you solve this problem and maybe you can
share your approach. Like,
I hope the problem is clear to everyone. Has everyone understood what the problem is?
Okay. So,
here is a
neural network based solution. So, in machine learning, we have this concept called
neural network, which is
basically a special type of program. It's ultimately a program.
It takes input and it gives you output,
but it is a bit different from regular programs (door creaks) or the programs you write
in, you know, C or Python or whatever. Like, we will also write this in Python, but we...
the way we write this program is a little bit different.
So,
if you see these,
this line here, this is basically our program.
Can you see? Y equals ReLU of,
uh,
this into this. So this at symbol is matrix multiplication.
Uh,
it's just, you know,
like A at B just means, you know, matrix A multiplied by matrix B.
(door creaks) So we take this matrix,
B mul- matrix multiplied with W2 and we apply this ReLU function on it.
So x here is the input.
So the input is this, you know, 60,000 photographs. We have converted this into a matrix.
Uh,
we will multiply this with a weight matrix called W1.
I will talk about quickly what all of these are, but...Yeah, x is our
input.
We will multiply the weight matrix. We will apply this ReLu function.
We will multiply that with w2. Then we apply this ReLu function again.
And y is our final answer. Like for each of these 60,000 images, you know, which,
uh, class are they in?
Okay. And
now I will talk about, you know, what is ReLu, what is w1, what's w2, what are all these
things. So
may, actually, no, maybe I'll first even just start with the input.
So x is the input, right?
(cat meowing) So x has dimensions n cross d.
If you've studied matrices, you know, right?
Like a
matrix has two dimensions, like, you know, its length and its breadth.
So n is the number of images. In this case, you know, n is 60,000.
So x is a 60,000 times 784 matrix. So there are 60,000
images and-
(sniffs)
... each image has 784 pixels.
Uh, yeah, one important thing here,
we are not storing the images in a 28 cross 28.
We're just storing it as, you know, single line of 784 numbers.
Uh, this is a very common thing that's done in machine learning.
I'm not going to talk about why.
(clicks tongue)
But yeah, that's just a thing that's done.
So
we... Instead of having, you know, dun-dun-dun-dun-dun, dun-dun-dun-dun-dun,
dun-dun-dun-dun-dun, like we have, you know, 28 cross 28, we're just going to
put everything in this one line. So 28 numbers, then the next 28 numbers, the next 28
numbers, next 28 numbers and so on. You know, you have 70, 784 of them in line.
And in the same way you have, you know, 60,000 such rows. So each row is one image.
There are, you know, 60,000 of them. So this is our input.
Then we multiply with this weight matrix called w1.
So w1 is this 784 cross and 800 matrix.
Uh, and what is w1? We will find out what w1 is. Right now, just assume it's a matrix.
We have somehow found out the value of w1.
Then we multiply it. Then we do this thing called ReLu.
Now, what is a ReLu? Uh, so here is the definition of ReLu.
ReLu basically means if you have a matrix for each cell, if it's a positive cell, just
keep it as it is. If it's a negative cell,
you remove that value and you put zero instead.
So if you had, you know, let's say, three, zero, minus two, four.
(metal clanks) So now this will become three, zero, zero, four.
So that negative value just got removed and we put zero there.
And all the positive values, we just kept them as it is. So ReLu is nothing special.
It just means remove all the negative values in that matrix and replace them with zero.
So, yeah, we
multiplied x with this w1 matrix. We made all the negative values
zero.
Then we multiply this with the w2 matrix.
Now w2's dimensions are 800 cross 4.
Wait, fuck. That's a mistake. It should be 800 cross 10. N- just give me a minute.
I'll just quickly edit that
because there are 10 classes, you know, 0 to 10. So this has to be 10.
Uh,
hmm.
Yeah, cool. Okay, so we have 800 cross 10.
And finally we get y, which is right now 60,000 cross 10.
Like if you do, you know, the dimensional analysis of this, you will get the answer, you
know, 60,000 cross 10. Like 60,000
times 784 cross 784 times 800 means 60,000 times 8, 60,000
cross 800.
Then you have 60,000 cross 800 times, you know, 800 cross 10.
That leave you 60,000 cross 10.
So now we have 60,000 images. And now why are there 10 dimensions here?
Like if... Think about it, there are 60,000 rows. In each row there are 10 numbers.
Why do we need 10 numbers?
So what you'll get in these 10 numbers is actually 10 probabilities
telling you which number is this. So let's say we had, you know, number called three.
We convert into, you know, 784 kind of thing.
And like this there are many images. At the end we will get...
For each image we're getting ten probabilities.
Now these are actually log probabilities. They're not even probabilities.
So here is an example of what we might get at the end.
And this is just for one image right now.
Actually, there'll be 60,000 of these, but for one image...
and
actually there will be ten of this. Uh, fuck, this is annoying.
I should edit that as well. Yeah.
(clicks tongue) I'm just going to make things easy for now and
go with LN0, you know, six times.
(keyboard clicking) Two, three, four, five, six.
Minus inf.
(keyboard clicking) One, two, three, four, five, six.
Zero, zero, zero, zero, zero, zero, zero, zero.
Yeah, I think that's about right.
Up- (door clanks) load this and here we go.
Ooh.
Yeah, cool.
So our y is actually... Here is what y looks like. So we have some values in here.
These are all run... numbers we get, you know, after doing the two matrix multiplications
and the two ReLus. This is what we, you know, get at the end.
I can close all this. Yeah. So we got minus 1.89, minus 0.28, minus
1.89, minus three,
minus infinity.And infinity just means, you know, very large numbers.
This could be in a minus, you know, 10 to the power of 32 or
whatever, you know, register we are using here for float numbers, so this is, like, the
largest number we can put in that register. So it's as good as infinity.
And y dash is what our actual answer is.
So remember, we know what the actual answer is, so here is y dash.
So now we're going to take y, which is our predicted answer,
and by the way, this y is actually log probability.
So what this thing actually means is
we are predicting, you know, 15% probability it's a zero,
75% probability it's a one, a 15% probability it's a two,
a 5% probability it's a three, and you know, 0% probability it's anything else.
Like, this is what this actually means, like, you know,
ln of 0.15 is, you know, -1.89.
So yeah,
this is the meaning of the thing but actually when we run this program, what we're going
to get is we're going to get this thing.
Uh, assuming we're using good weight matrices, W1, W2.
We have not yet told you where did we get W1 and W2 from, but if you're using a good
value for W1 and a W2, we're, we might get something like this.
Or we might get something different.
It's just this thing is saying, you know,
this character is probably a one because, you know, it's saying there's, there's a 75%
probability it's a one.
And now,
yes, here's the predicted, here's the actual thing.
Now there is a question of, yeah, how do we find good W1 and W2?
So now here is an important question.
And again, I'll,
I know I've
given you a lot of information. I want you to just think for five minutes, you know.
Here's the program. We want to find the W1 and the W2.
These are the dimensions of W1 and W2,
and
we want to ensure at the end,
these predicted values should match close with the actual
values. So like if we manage to get exactly, you know, the exact actual values, you know,
here, ln0, ln1.
What, what is ln1? Ln1 is zero, right? Yeah.
So if we could get, you know, minus infinity, zero, minus infinity, minus infinity, minus
infinity, minus infinity, minus infinity and so on, like,
that is our ideal answer.
And you know, otherwise anything else is a bad answer.
Also, we have this measure of how bad is our answer, like
our prediction is not a hundred percent accurate. There is some inaccuracy here.
So how inaccurate is it? So that thing is called loss.
So thi-
this which tells you how far away is your prediction from the truth.
So this is defined typically, there are multiple ways to define it, but common way of
knowing it is
negative sum of y dot y dash.
So these are our predicted log probabilities.
This is, you know, our actual ground truth.
We are going to dot product this. So dot product means what? Just this one we're taking.
So this,
we just took this.
And
yeah, that's our loss. So
what probability did we give, you know, the correct answer.
In this case, we are just keeping it simple. We're just keeping that as a loss.
In practice here, there are other loss functions also.
But
yeah, for now it's just the log probability of the correct answer.
That is our loss.
So we want this to be less. And remember this, we have now for one image.
We will add this value up for now 60,000 images.
We will get, you know, 60,000 such values.
We will add them all up
and that tells us, you know, how inaccurate our thing is.
So yeah, I, again, I hope this whole problem setup is
clear. Now we need to figure out a good W1 and a good W2.
Like how will we search for a good W1 and a good W2?
That's the question here.
Okay. So
how, uh, how many options do we have for W2?
Like, we need to search for a good W2, but like what's the search space?
The search space is this, you know,
784 cross 800 search space. 784 cross
800. That's
627,000 values, right?
That is also wrong.
627,000 values and W2 has
8,000 values.
2 to the
power...
Yeah.
Huh. Lot of mistakes. Yeah.
Yeah. So how many values do we have? We have...
Here we have around...
This is not changing. Cool.
784 times 800. We have around six to seven...
Around 600,000 values basically. And
W2 has, you know, around 8,000
values.That's
60 s-
sorry,
784 into 800 into 800 into 10.
(keyboard clicking) This is...
That was in... Yeah, this is around 5 billion, I guess.
2 to the power 5... That means 75 into 10 to the power of 9. (keyboard clicking) Cool.
So we have 2 to the power of 627,000 into 8,000, that's around 5 billion values.
Uh, so, we have... Even if in each cell we can only set, you know, 0 or 1, and actually
we can set values in between also. We can set, you know, 0.3, we can set 0.5.
We can set, you know, 50,000. We can set, you know, -87.5. You know, we can set anything.
But even if we could set only 0 or 1, we have, you know, 2 to the power of 5 billion
possibilities. And in practice because we can set many values, we have actually a lot
more possibilities than that. So, we definitely cannot do any sort of brute force.
We cannot even do, like, a very smart sort of brute force.
Like, even if you reduce this, you know, 2 to the power of 5 billion into, you know, 2 to
the power of 1 million possibilities, like, maybe you find some symmetry and you say,
"Okay, these cells are similar to these cells, so we don't have to actually search them
all together. Let's just search these ones and we'll copy the values in there." Like,
often when we have brute force problems, this is how people, you know, make it efficient,
right? They try to reduce the search space.
But even that's not really
going
to work here because our search space is just so crazily large.
So we can't really do it. We have to find something better.
And also,
actually, if you look at this
problem more, you will understand there's also not, like, an exact answer.
Like, you can't
just look at this formula and calculate what they look.
Here is the formula for calculating,
you
know, the optimal
W1 and W2. Like, to optimize loss, you know, here is the optimal W1, here's the optimal
W2. There is no direct formula to do this either.
Like, just by looking at it you won't understand this.
If you spend maybe a month (laughs) just
thinking about this problem, you'll realize, like, yeah, there's no way you're getting an
exact formula for this. So what does that leave us with?
Remember, we need to optimize loss given a W1 and W2.
So, we are going to bring in a concept from calculus.
Uh, in calculus, when you have to optimize one variable with respect to other variables,
what do we do, typically? Yeah. So we take, you know, d loss by dw1 equals to 0.
Like you remember, right? Maxima, minima. Yeah.
So, that's kind of what we're going to do. Uh, there are a couple of complications.
The first complication is w1 is not a number, it's a matrix.
Because it's a matrix, you know, that complicates things a little bit. The...
Because, you know, if you just do dL by dw1 and d...
If w1 was a scale of value, then, you know, we can just do dL by dw1.
W1 is a matrix, so if you do dL by dw1, now that is going to be another matrix.
And you will say, "Okay, well, wh- why can't we just put that matrix equal to 0?" Like,
that mat-... dL by dw1 is a matrix, we'll calculate it.
We'll say, you know, that matrix equal to 0 matrix.
And similarly,
you know, dL by dw2 will be
one more matrix. We will get that matrix, we'll equate that to 0
matrix. We will solve this entire system and we'll get the answer.
And, like, first of all you can try to do that.
It's still going to be a very large system to solve.
Uh, if someone wants, they can try this out. I have not tried it. You can try it out.
But, y- like, you can imagine, right, you're going to be getting...
Like, again, you're going to get, you know, a system of five billion equations.
Like,
five
billion
linear equations, and you're going to try to solve them, I think, and...
Well, okay, you can put it, you know, in a linear algebra form, maybe you'll get it
solved, but, yeah, that's again hard.
And that's even more not going to work when you make...
like, when you add more matrices and you make this thing bigger.
Like, instead of, you know,
w1 relu, w2 relu, what if I then did, you know, w3 relu, w4
relu, w5 relu? Like, you know, multiply this with another matrix, take a relu,
multiply that with one more matrix, take a relu, multiply that with one more matrix, take
a relu.
And in practice, actually, for more difficult problems, we need to do that.
Like, right now we have picked an easy problem, so this gets solved pretty well just
with, you know, two layers, like two matrix multiplications and relus.
But
for
harder problems we might take more. So,
you know, for transformers, like, you know, your latest AI use, like, GPT 3.5,
GPT 4 or whatever, you might take, you know, like, 80 layers or even more.
So there will be, you know, 80 matrix multiplications running one after the other.
And then you'll be like, "Well, how do we find the optimal
matrix?" Like, you're not going to be able to solve, you know, d loss by dw1
equal 0, like, analytically.What you can do...
Is this thing called gradient descent, which is
basically...
Yeah. Basically,
move along the direction
decided by the slope. So, wherever you are right now, so I mean, I can make...
This might be easier to draw with, you know, a figure.
But
whatever your loss value is right now,
find out dL by dW in there
and whichever direction it's pointing to, just move in that direction
because that is... If you move along that direction, that's where the loss will reduce
the fastest.
So, this is the important trick in gradient descent.
So, we are going to find dL by dW1 and dL by dW2.
We'll move in that direction and we'll find ano- new W1 and W2 that are slightly
better than the previous ones.
And then we can repeat this process. You know, once we have W1 and W2,
we'll find the gradient, we'll move a little bit to find a slightly better one.
We'll find the gradient again. We'll find another W1 and W2 that are even better than
that. We'll repeat this, repeat this, repeat this.
You know, after enough times repeating this process,
we'll get a really good W1 and W2.
So for this, we need to find dL by dW1 and dL by dW2.
This is... Now actually finding these gradients is kind of straightforward.
It's also kind of boring. Uh,
if you have, you know, half an hour to
sit with a pen and paper and compute these gradients, it can totally be done.
So I mean, dL by dW2 is basically
dL by dW2 IJ for each of these IJ.
So it's like, you know, we had matrix times matrix ReLU matrix ReLU.
So now D
and O... So this is Y. And then we did, you know, dot product with the actual answers, we
took a sum.
So now this D loss by...
We have to do with each of these values.
If this value changes, you know, how much does the loss change?
If this value changes, how much does the loss change?
And so on for all the values in this matrix.
That's dL by dW1. Same way we look at this matrix.
We'll be like, if this value changed, how much should the loss change?
If this value changed, how much should the loss change?
And so on.
We'll get one more matrix of those changes. That's dL by dW2.
And yeah, those are just your gradients.
Like,
if you change each, if you change this...
Like for each of these values in W1, if you changed it by some epsilon, it will change
by, you know, this much times epsilon.
Like, you change it by epsilon, you change it by maybe 0.03 epsilon.
So 0.03 is, you know, your gradient.
Yeah. I mean, I can maybe find a nicer way of writing this.
Just give me a minute. I will
write this in a kind of nice way.
Like, uh...
The loss is equal to...
Into, you know,
something.
Then we took, you know, a ReLU of the entire thing.
Let me just use a... This bracket to differentiate, to
keep it, you know, looking separate.
And then we're gonna multiply that with yet another matrix.
And
then we took, you know, yet another ReLU.
So I'm using another bracket for that.
Then what did we do? We multiplied this with the actual answer, correct?
Yeah.
So this we multiplied with
w dot y dash.
And then what did we do? We took a negative sum of the entire thing, right? Yeah.
So sum we can just, we can just say, you know, magnitude of this thing.
Sum, sum... Yeah, I'll, I'll just use the magnitude
notation.
So here we have, you know,
X00.
This goes all the way to, you know, XND,
and-Here
we have w1,
0, 0.
(background noise)
And this
goes to
w1 or w1 dimensions.
W1,
D, D1.
Instead of calling this D1, let's just call it h.
I think h is more convenient.
Yeah.
So we have w1 Dh.
Then similarly we have, you know, w2, 0, 0.
That goes still,
yeah,
w2 Hc.
Yeah. I think that looks good.
Uh,
yeah, just give me a minute. I need to
get that looking.
(background noise)
Yeah.
(background noise)
Where are we? This is not yet updated.
(background noise) Yeah, this looks good.
Yeah. So we just want to find new dL by dx.
No, sorry, Dw1 0, 0.
Yeah. And remember like these are partial derivatives.
Uh,
I hope everyone remembers, you know, partial derivatives versus total derivatives.
Uh,
partial derivative basically means when we are doing dL by Dw1 0, 0, we are assuming
everything else is constant. So now all these values are constant, you know, everything
in x,
all the other values in w1 itself are constant, all the values in, you know,
w2 are assumed constant. You know, I true is assumed constant.
So,
you know, x0, 0 is the... Sorry, w1 0, 0 is the only thing that's changing.
We change this by, you know, an Epsilon.
(bell ringing) We're just checking by how much will L change,
by what multiple of Epsilon.
So
now this we can actually solve, you know, using chain rule.
So we can write it,
you know, this way like...
Wait, that's bad. I can't use H here and H here as well.
I will have to change this to something else.
Uh, (metal clanking)
(background noise)
let's just make this small h. I think that looks good.
And yeah, this will become a small h.
You know the notation that we have stepped is below as well, so I'd rather not change
this.
Mm.
Yeah, I think I'll just have to make this small h then. Yeah.
Yeah, but that's kind of an annoying way of doing this.
(background noise) Just give me some other letter.
Uh,
e? Okay.
I, I don't know. This is probably not what's used in, you know, the standard textbooks or
whatever, but I'm just going to make this thing e, you know, fuck it.
Uh,
yeah, let me just check, is h sitting anywhere?
This case-sensitive.
Yeah.
(background noise) Cool. Yeah, cool. This is good.
(background noise)
Yeah. So we just made that number of dimensions E.
So E is, you know, 800.
This is why they called, you know, 800 hidden layers.
Like if you ever go see, you know, the...
Some like standard lecture on this, this is a very standard problem, you know, two layer
fully connected network trained on MNIST.
Uh,
this thing will be called this number of hidden layers2.
So there are 800 hidden layers
in our neural network.
So you can solve this. If you solve this, you will get,
uh, this answer. So
this is gradient W1, you know, dL by dW1 is this answer, and
uh, dL by dW2 is this answer.
So basically,
this x dot T is x transpose. I'm just clarifying, you know, the notation.
And maybe I'll just start here. h dot T is, you know, h transpose, uh, B hat.
Uh, here we are taking minus Y true, so minus Y true times mask.
Now mask is
all the positive values of a2. ...
a2. Okay. a2 is h times W2.
Can you see? It is a little bit complicated, but yeah.
Like let's just go back here.
So h times W2. So basically they took this thing,
like this, this much is h times W2 down here.
They checked which of those values are greater than zero,
and s- some of those are one and some of those are zero.
So we made that, you know, a mask, mask2.
And by the... Why did we end up with something weird like this,
where put... the positive values are made one, the negative values are made zero?
It's basically the
positive values, their gradients are one, and the negative values, their gradients are
zero. Like, this is the gradient of ReLU.
Like, if you have a function that goes, you know, ting, ting, so now the gradient of
these values is zero, and, you know, the gradient of these values is one.
So that's why you ended up with this thing called a mask.
If you would actually try deriving this, you'll understand, I think.
I don't have time to, you know, derive the entire thing right now.
The...
So
you'll, you'll get a mask, you'll do minus Y true into this mask.
Then you'll do h transpose matrix multiplied with this entire thing, that gives you your
gradient.
Then now that you have this gradient...
okay. I think that you keep separate, and separately you also need to do this calculation
for, you know, gradient of W1.
So this derivation, I'm going to leave it as homework.
Uh, those of who, you who are good at linear algebra, you can probably get this
derivation done. It's basically, you know,
here is the original thing we started with.
Assume you know, except W100, every, just keep everything constant.
Then except W101, keep everything constant.
Except W102, keep everything constant, and just find dL by, you know, that value.
So you'll get these derivations. You know, we have found the matrices.
So, but this works at one particular point.
So,
you know, access constraint we have assumed in our input, provide you through the actual
answers is constraint.
W1 and W2 are the things we need to find.
So if we are given a current W1 and W2, we can find another gradients at that
particular position.
Then
we will update W1 and W2,
like we multiply these ingredients with a small epsilon, just a tiny amount.
The actual number is something we can decide later,
how big we want epsilon to be. That is a whole separate topic.
Uh, there is this thing called an atom optimizer that is used nowadays.
I will get to that later. Not right now.
But right now, just assume E is some very small constraint value.
It could be, you know, 0.0001 or whatever.
So we multiply that with our gradient and we update W1 and W2.
So we get as, uh, a new W1 and W2, which has slightly less loss.
The L now for this new W1 and W2 is a little bit less,
or well, sorry, I think L is negative, so
L will go up actually, yeah.
And then we repeat this process millions of times.
Like, we will have to do
these calculations, we'll have to do them millions of times and keep updating W1 and W2,
and at the end we'll get a good answer.
Uh,
yeah.
This is what
a two layer fully connected network
is.
And how did we manage to run this? Uh, we will probably need a GPU to run
this.
So the GPU is basically
a
chip that does matrix multiplications very fast.
Like, that's the whole thing.
The GPUs were actually way back, you know, before deep learning was very popular, GPUs
were actually designed for gaming.
And for gaming you need to render like, you know, vectors into
graphics.
So your vector might be, you know,
draw a line from 0, 1 to 4, 3.
And we need an actual, you know, pixel matrix.
Like we need all, each of those pixels, you know, set to 1, 1, 1, 1, 1, 1.
Like, even right now, if you look at this entire thing,
like what's showing on your monitor, actually,
initially the instructions are like, okay, you know, draw a rectangle here for this tab.
You know, put some text inside here. Now what the hell is text?
Now go and get this, you know, font sheet from here.
There is some, you know, bunch of images for each of these letters....
you know, draw
lines here,
draw, you know, a rounded rectangle here and, you know,
this gets a lot more complicated when you are playing a game, for example.
So when you have a game, you have, you know, a 3D game engine,
so you need to convert all those lines into, you know, this ultimate 2D view.
So doing all those conversions also in practice you end up doing using a bunch of matrix
multiplications, so
that's what GPUs were originally designed for.
Today, you know, GPUs are just being used for AI and people are even
redesigning GPUs to become better at AI. So, yeah.
So that's why, you know, GPUs were originally good for matrix multiplication.
And if you look at these, you know, the only things we have mainly are only
computationally expensive operations here are matrix multiplication.
We also have this, you know, zero check, is this positive or negative like across the
matrix? This can also be parallelized and GPUs can parallelize this as well.
Like, again, remember these are like millions of values there.
A2 is,
uh, how big is A2?
60,000 cross 10, right? Yeah.
Yeah, so this is 600,000 values. We would rather, you know, check all of this in
parallel, not check them, you know, one by one.
So,
yeah, all this can be parallelized.
We are not doing any copy-paste anywhere and that is important.
If we had to, you know, copy-paste, you know, one million values from here to there, that
would be a problem, it will become slow.
And because we're doing this on every iteration, so like, we have to copy one million
values, you know, one million times.
So that's like a trillion copy-paste operations and that's bad.
So, if we had copy-paste that would be a problem.
We don't have that either.
We just need to in place, we need to update, you know, w1 and w2.
So, yeah, this will run quickly on a GPU.
And,
yeah, okay, there's a note, I have simplified some steps here.
So one thing which I've simplified is, in practice we are not going to be...
typically, we are not going to be processing our entire 60,000 images all at once.
We will typically split it into batches.
We will do, like the gradient descent we'll be doing, we'll be taking different batches
in different iterations.
And this you can actually maybe go read online how batch SGD works but, basically we su-
spread the 60,000 into batches of let's say 100 images each.
We compute the gradient not over the entire 60,000 images, but over 100 images.
So our x,
when we do this anti-gradient calculation, we will take x as, you know, just
100 cross, uh, 784 instead of, you know, 60,000 cross 784.
We'll compute the gradients, we'll update w1 and w2.
Now we will take a different batch, and th- here's where things get weird,
uh, and then we will compute the gradient on that.
But now you will ask, "Uh, isn't that a whole separate problem?" You know?
Like the optimal w1 and w2 for this batch and optimal w1 and w2 for that batch are
different like,
this is an entire separate gradient descent problem, that's now a separate gradient
descent problem but, we'll take one gradient from here, we'll take one gradient from
there, we'll make a third batch, we'll take another gradient from there.
Then we will use all of these gradients, we'll add up all of them, you know.
Epsilon plus this, epsilon plus this, epsilon plus this, and we'll keep doing all of
this.
So this is called, you know, batch SGD. SD is stochastic gradient descent.
And here there's a
bit of randomness that will come in because we are not taking all the images at once,
we're like just randomly selecting
different batches and taking gradients from all of them and adding it up.
There's a bit of randomness that comes in, but this will also give a good enough
solution.
Yeah, and by the way, the only reason we didn't take all of them is, again, 60,000 images
might not fit in your GPU advance, and that's the only reason, like, you...
If you can fit in 60,000 G- like images into the GPU you might be honestly better off
just doing the normal gradient descent, like, the regular way of doing it what I've
described, but
if you had even more
data like instead of 60,000 images you had, you know, like a billion images,
and in modern datasets you have literally, you know, billions of images, billions of, you
know, text documents, so,
you can't fit all of that into GPU, so, yeah.
Then we use an optimizer called ADAM.
ADAM is basically just a way of changing these epsilon values, so, we
start with big epsilon values like we first descend f- fast and then we reduce the
epsilon values, so Adam is just, how do you do this?
And
the main reason to do this is just to again save cost like
how do we do this anti-gradient descent more quickly?
And last what? Cross-entropy loss,
yeah I think I've already defined this using cross-entropy loss only.
But yeah cross-entropy loss is basically this thing about the log probabilities like we
didn't use actual probabilities we used log probabilities like that,
the loss that comes out at the end of this is called cross-entropy loss.
Yeah.
So now,
here's some homework.
So the first homework is I have already told you, you know you need to
derive this backprop formula yourself.
This thing just actually derive this or at least go online search for a
derivation you know look at it like you can ask O3 to do this stuff for you like
actually O3 computed this you know you can ask O3 give me more steps you can be like you
know O3 I do not understand this you know explain this or whatever.
Like I can literally show you what I'm talking about. Uh...
Explain this
in about
10, 10
to 15 minutes.Like I ment-
mentioned a point earlier, you know, like...
Wait,
let's, uh, copy this.
(keyboard clicking) And put
this in here.
(keyboard clicking) Why are we doing
A2 greater than zero?
(keyboard clicking) Like can you see how we did A2 greater than zero?
I'm just asking O3, you know, why are we doing that?
(breathing)
(chair creaking) So, you can ask for re- these types of, you know, specific questions.
Yes. What does it say? Okay, it just... yeah, this is just defining the problem.
Okay, there are... chain rule.
Yeah, that's an lengthy-ass derivation.
Uh,
yeah, here is our explanation.
Why is A2 greater than zero?
Yes, a ReLu is this function, you know, zero if negative, you know, Z is
positive. And
its derivative is zero and one.
That's why we did this.
Yeah, I mean, you can sit and read all this if you, for instance want to understand, you
know, how to do the gradient of ReLu.
So you can... yeah, that's your first homework.
The second homework which is more advanced, but again, you will find this online.
You don't have f-... You can do this from scratch if
you really want to understand this or, you know, you're a bit of a masochist, uh, but
you will find this assignment existing online too, which is, you know, actually train
this fully connected network with, you know, 800 hidden,
uh, dimensions.
Use Pytorch. Uh,
you will get a 1.6% error rate, which means in out of the 60,000,
uh, examples, not... (keyboard clicking) Sorry, on the remaining 10,000, right?
(keyboard clicking) So on the remaining
10,000 you'll get,
(keyboard clicking) uh, what am I doing? Yeah, 98.4% correct.
So
9,840 examples you'll get correct. And about 160 you'll get wrong
on the remaining 10,000 examples which you've not seen till now.
(chair creaking) So that's pretty good.
(child talking in background) So yeah, that's what
you will do. (paper rustling) Yeah, and also I want, if you're doing this
assignment, like you're actually sitting and writing this in Pytorch,
uh, I would recommend you actually write down these gradients.
Otherwise Pytorch has this feature called autograd,
which means Pytorch can compute these gradient formulas on its own and use them.
Like,
this is part of the reason why NVIDIA is on track to become a trillion
dollar company,
uh, is, you know,
Pytorch does these types of calculations on their own.
So ML engineers don't need to sit and, you know, break their head on how to do
gradients,
like how to calculate this gradient formulas.
This is not the only reason NVIDIA is so rich, but, uh, you know, it's part of the
reason.
(paper rustling) Uh, so yeah,
I would recommend for the homework assignment, you know,
(child playing in background) uh, yeah, use these actual formulas which are written
down here. Don't, you know,
ask Pytorch to i- autograd to, you know, just do it.
Cool.
Any questions? I think
this- (bell ringing) what we've done here, this is actually the most important thing in
the lesson today.
The remaining one hour is a bit less important (laughs) than this one hour.
If you have just understood this very well, you have understood, uh, like a lot about how
deep learning works.
Cool.
Now (coughs) comes the next section.
Yeah.
So here is a question.
Why did we do all this, you know? Why did we use two layers? Why did we use a ReLu?
Why did we define our loss function this way?
Like we could have defined the loss function different ways, you know, actually if we
wanted to.
Like
here I took, you know, this value and ignored all the other values.
And also I said this should be a log probability.
Like why did we decide all this, you know?
And in practice, when you go into proper deep learning, you will get even more questions.
Like you will learn things, you know...
There will be a lot of these types of tricks.
Like okay, use RMS norm here, use Softmax here,
define attention layers in this way, use these many layers, like we use two layers, you
know. Why two layers? Like...
And this in practice becomes a lot more layers.
You know, what's a residual layer? Like, we are going to encounter a lot of tricks
and whenever a question comes in your mind, you know, why are we doing this trick?
Always by default, here is the answer.
Um, I'm going to just open the answer right now for you.
(person coughing)
(water running)
Should we bow?
Yeah, he's a king. Seems like I'm always thanking you for something.
(clears throat) What are you doing?
Uh, we, we don't do that here.
Yeah.
So that's the answer.
We do not ask these types of questions here.
Do not ever ask me, "Why did we do this?" and "Why did we not do that?"
The answer is
some guy tried it a few years back, it worked out.
Some guy tried a little bunch of similar things, those did not work and this worked
because who knows? And that's the answer. That's always the answer.
You're not going to get no- a better answer than that.
If you think you have a better answer than that, often you're actually mistaken.
And that comes here. Like lots of people will give intuitions for why, you know, this
trick works and that does not work.But often, these intuitions are created
after we have actually tried the experiment.
Like,
we tried an experiment,
some particular trick worked, you know.
Let's say this tool ReLU worked very well.
Instead of ReLU, we could have refined a different, you know, activation function.
But ReLU worked very well and now, let's...
We will now invent some, you know, intuition for why ReLU is a good idea and why, I don't
know, why ReLU is a bad idea or whatever.
Uh, yeah. ReLU is the real thing. Uh, I can show you.
Like, there are other loss functions.
Yeah. So this is what ReLU looks like. Here's the formula for ReLU.
ReLU is just, you know, negative X, make it zero, positive X, make it, you know, X.
And here is a bit more ... Yeah.
So, yeah, people just
create their intuitions afterwards.
Also, yeah, especially modern day ML, because there's only a few labs that are doing, you
know, the latest research,
uh, only people in those labs are going to have these intuitions anyway.
Like,
they are the ones who are running hundreds of runs.
Like, to get an intuition, you need to try some things and see that some of them work and
others did not work. Like, it's a very empirical,
uh, field. It's not a theoretical or research kind of field.
It's just pure, tried this, it worked, tried that, it did not work.
And right now, the people trying things are only, like only if you have, you know, a
company that has raised, you know, millions of dollars in funding, you can try things
anymore. So,
they are the only ones who have these intuitions anymore.
Yeah. So if you ask me, you know, why all these older things?
So,
here are, you know, older architectures, here are, you know, older activation functions,
older loss functions, older optimizers, you know, older regularization
tactics. You know, a lot of these things, you will read about these in, you know, five
year old tutorials. You will not read them, about them in today's tutorials.
And
the only reason is just, you know, we found something else that works better.
That's the only reason.
Cool.
Now, we are coming
to
modern day
machine learning, deep learning, you know.
We are coming to transformers.
So we have, you know, ChatGPT or Grok-3, or you know, Claude.
Any model you are using today is basically a transformer.
Uh, here is the lecture from which I first studied transformers.
I still think it's an amazing lecture.
It's by this guy called Justin Johnson.
It's four years old, but I think it's still relevant, even despite being a four year old
video. You know, four year old videos would otherwise be considered ancient in
deep learning speak. So if you have a video that's four years old and it's still relevant
today, you know,
that means it's a good
video.
So yeah, here's lecture 13. This will make more sense if you actually see some of the
previous lectures. It is going to make less sense if you just jump into it directly.
But yeah, if you want to understand transformers more in depth, that's the lecture for
you.
And here is an actual transformer we are going to look at.
This is LLaMA-4. It was trained by Facebook, uh, as the model
card.
Yeah.
Llama-4 Scout, LLaMA-4 Maverick.
Hmm. So,
yeah. Scout has 109 billion parameters, Maverick has 400
billion parameters. I'll explain later on, you know, what is parameters.
Right now, I'm just saying,
like, this is basically the size of the model.
Like, it's the size of the weight matrices.
You know, we talked about, right, like W1, W2, so.
Transformers also have weight matrices.
This is the
sum of, you know, all of the weight matrices, in total, how big are these matrices.
So there are a total of 400 billion numbers inside all these weight tr- matrices,
basically.
And yeah, this was released just two months back, in April 2025.
I think I'm going to have water.
Okay, cool. Uh, this was trained on five million GPU hours.
It basically means you took a GPU, you ran it for, like, 500 years.
500 years, right?
Five
million hours
is... Okay, yeah, 570 hours. So,
you take, like, 570 GPUs and you run them together for a year
or, or, you know, you can take more. You can take, like, uh, like, what, 7000 GPUs and
you can run them for a month.
Uh, like,
a GPU looks like this, by the way.
Yeah. Here is what it...
GPU looks like.
It's just a card like this.
And the cooling is, I think not... Yeah, this includes the cooling.
Like, separately you need, you know, these are the GPUs inside and you need the cooling
unit separately.
So, yeah.
So now I'm going to actually look at the code.
The code is not that too difficult to understand.
If you already have studied transformers, you might actually understand a fair amount of
the code.Yeah.
So again, I don't have time to go through the entire thing, but yeah.
See here, this is called attention. This thing called attention is the most important
part of a transformer.
Uh, here we defined forward.
This is a forward pass, so
you know, what we do to our input to get output, that is called a forward pass.
Here is forward for the transformer block.
Again, I will explain all this soon.
Yeah, I think this is where our actual...
Yeah. Okay, so this is LLaMA 4.
And where is LLaMA 4 forward?
Okay, LLaMA 4 inference mode. This is where we generate... Ah. Yeah, transformer input.
Okay. Yeah. Where is this line called? 93.
Yeah, okay, model equals transformer model args.
So here's the input we give to the transformer and we... No, no, no.
Here's the
hyperparameters
and I
won't explain that right now.
Yeah, I think this script is less useful. It's this one that's useful.
Yeah, return transformer output.
Trans...
Oh, yeah, yeah, yeah. This is useful. Very useful.
(sniffs)
So this is where we start. We start with some tokens.
Okay, so we got self model input.
Tokens equals model input or tokens. So here is already a first step that has happened.
Ah.
Where are we?
Yep, so here is a bunch of steps that happen in a transformer.
The first step is called tokenize. We use a tokenizer.
And tokenizer is pretty straightforward. Basically means we...
Actually first, sorry, I should maybe start with, what the hell does a transformer do?
So a transformer takes a sequence of tokens as input and it gives you log
probabilities about what the next token should be.
So
tokens are just words, you know, like
I'll say, "My name is Samuel. What is your name?" So my is a token, name is a
token, is is a token, Samuel is a token.
There's a sequence of tokens and
a transformer will be given, you know, a sequence of tokens, you know.
"My name is Samuel. What is?"
Predict the next token. And it will give you, you know, log probabilities of like
what is a probability, what is your probability, what is my probability,
what is their probability. Like you will get, you know, a sequence of numbers giving the
probabilities for every possible next token.
So input is some tokens, output is some tokens.
So
yeah, first we convert our words into tokens.
The tokenizer just says instead of my there is a dictionary of maybe some 100,000
popular words in English and they're all numbered, you know, zero, one, two, three, four
till 100,000. Instead of my, maybe my is, you know, word number
4,058. So instead of my we'll put, you know, 4058.
Okay, my name, maybe name is, I don't know,
50732 word or it's the 50,732nd word.
So instead of
this we'll put, you know, 50732 or whatever.
Yeah, actually we won't put 50732. We will
fill 100,000 zeros and on the 50732 index instead of zero we'll put a
one. That's actually how we will fill the matrix.
But yeah, remember we need matrices. Our entire...
All this work has to wor- do- get done on matrices. We can't do work with words.
We have to start with some X. X has to be some matrix input.
So we'll convert our words into, you know, a matrix like this.
So that's just what tokenization is.
Then next step, we'll have this thing called a positional embedding.
Where's the positional embedding?
Yeah, start position.
Okay, so we got a position. What did we do with the position?
We converted that into frequency.
What did we do with the frequencies? Okay, we took the frequency here and
we defined layer.
Like, you know, if you have to sit down and understand this code, yes, it will take time.
I'm going to do... Take a shortcut. I'm going to copy this entire code and I'm going
to ask our AI to tell us
what does everything here mean.
Explain
all the steps in simple English.
Bam.
Yeah, uh, O3 is a great tool for learning anything by the way.
Like O3 sometimes makes mistakes, it can understand it.
And that can be a problem if you're a beginner.
But apart from that, O3 is a great way of learning anything.
Cool. So we
have a huge number of steps, you know.
Utility pieces. Let's skip the utility pieces right
now.Yeah, top level transformer.
It stores hyper parameters, I'm going to ignore that. It...
Yeah. Yeah, this is the thing we actually want, this is the forward pass.
It converts tokens to embeddings.
If there's an image, we will also convert that into embeddings.
And right now I'm not going to talk about images, let's just consider text for now.
But we can also convert an image into embeddings and then we do that right at the
start, and then, you know, rest all the steps there on the same for text or images.
Yeah, we
put these positional embeddings, which is basically along with the token we also kind of
store
which position it starts instead of, you know, my name is Samuel, it becomes like my
one name two is three, Samuel four.
So even if you're in the middle of the sentence somewhere you are at like Samuel four,
you already know you're at the fourth word.
Otherwise, when you're somewhere in between, you
may have forgotten where exactly are you.
Yeah, then this, this attention mask,
I'm going to skip that.
Yeah, I will, I will have to ask this a bunch, bunch of more questions to actually
simplify this. Like, you can ask or refer more questions, you know, what the hell is
this? You know, what is this? You can keep asking, it will eventually simplify it, but it
will take a lot of time.
I have actually already simplified this stuff, and
I'm going to follow that.
So here is the simplified version.
See, we tokenized, we put positional embeddings, we got image embeddings.
Then we have N of these, so this entire thing actually repeats, you know, maybe 32
times or 64 times or 80 times. So let's say we repeat this 80 times.
We do something called an RMS norm, we do something called multi-headed attention, which
is the most important part of this entire model.
This is basically with our input, we, you know, computed something, we computed
something, we computed something, we computed something.
We did this entire thing 80 times, then we did a projection and we got an answer.
So we did something called a multi-headed attention, we did add residual, we did RMS
norm,
we did a feed-forward layer, we did add residual.
So this is the entire thing we did,
and I don't have time to explain all of these in detail, so I'm just going to explain the
steps which are really important.
Steps which are important and the steps which are simple.
So the steps which are simple are like tokenizing is simple, embedding is simple, RMS
norm is simple,
residual is simple. So I'll just explain some of these which are, like, very simple,
and the thing which is important. So the important thing is this projection
and this, uh, dot product attention.
So I'm, I'm going to explain some of these more and some of these less.
So anyways, keeping the big picture, we started with the sequence of tokens,
we did this giant amount of calculation on it,
we got output, you know, log probabilities for what the next token should be,
and this entire thing is described in this, you know, 500 lines of code here.
If you go through this, like most of these steps are all, all the code is just sitting
here. Some of it, a little bit might be setting in ffn.pie for example.
So here this, you know, another 50 lines of code but,
yeah, the vast majority of the code for this entire process is just sitting in this one
script. If you actually
sit and read this properly you will understand everything I've talked about here is
actually sitting in there.
And okay, a bunch of these steps are all actually optional, like rotary embedding,
optional, headwise RMS norm, optional, temperature tuning for long context, this is
optional. So you can actually, you can even kind of ignore these for now.
These are not important.
Cool. So we started with the sequence of tokens, we did all sorts of
computation on it, we got an output, now what do we do with this output?
With these log probabilities?
Remember right now, actually these log probabilities are completely inaccurate, they're
like garbage,
because we don't know what our weight matrices are.
So like when we do these calculations we will need weight matrices.
So like
we will need a weight matrix here for embeddings, we will need a weight matrix for the
projection,
we will need a weight matrix for this projection,
we will need... Actually we'll need three weight matrices here,
we will need weight matrices for the feed-forward layer.
So there's a lot of weight matrices here which right now we don't know,
and we're just going to initialize them with some junk values.
And yep, we are
going to do gradient descent remember? Yeah.
We are going to define some loss at the end and we are going to do dL by dW.
So first of all, what is loss?
Loss is how much our predicted answers differ from the actual answers.
So we will get some log probabilities of the next token,
and we know the actual next token. So right now with that data we are assuming we
actually know the next token.
So now this, given this and this, we can mul- dot product these two, we will get a
cross-entropy loss. This is just a value telling us, you know, how bad is our prediction,
which right now when we start off it's pretty, pretty bad.
Then we compute, you know, gradients,
you know, dL by dW, and there are a lot of Ws here.
There's, you know, this projection W, there's, you know, the dot product W, there is a...
No, sorry, here there's no W, there's,
yeah, this, this W, there's, you know, tokenizing W.
But yeah, the most important weight matrices are actually this one, the one in the
projection step, and multi-headed attention.
When I describe multi-headed attention I'll talk to you about what this projection step
actually is.So,
anyways, there is a weight matrix that is required for this.
We will do
d loss by that W. We will then get, we will then update that
W.
We will set a good epsilon. Remember epsilon?
Uh,
epsilon is basically once we know the gradient, how much do we nudge our weight matrices,
right? Do we nudge our weights a little bit or do we nudge them a lot?
So, we will use Adam with weight decay to
decide how much to update our weights.
And then, we will spend hundreds of billions of dollars, uh,
running this gradient descent, just updating W, update W, update W, and you spend
hundreds of billions of dollars this way.
Uh,
yeah, this might distract if I bring this up right now. But actually, I'll just...
Is my note not working? What? Hmm.
Yeah, my note is fine. It's slow.
Okay.
(instrumental music plays)
Stargate, put that name down in your books, 'cause I think you're gonna hear a lot about
it in the future. A new American company that will invest 500 billion dollars
at least in AI infrastructure.
The data centers are actually under construction.
The first of them are under construction in Texas.
The Abilene location, which is our first location.
Abilene is a town on the western central plains of Texas.
We kinda scratch and fight for everything good that comes our way.
I had people reaching out and they said, "When is the President going to come see you?"
And I said, "Well, he hasn't texted me yet, so."
(laughs)
I'm thrilled we get to do this in the United States of America.
I think this will be the most important project of this era.
Welcome to Abilene, Texas. This is the site of Project
Stargate, a very mysterious, much talked about, much speculated about project,
that brings together-
Yeah, so by the way, this is a documentary on the
largest computer data center being built in the world, as of, you know, June
2025. Uh,
you know, one year later this data might be outdated, but as of today it's the biggest
one, uh, under construction.
For OpenAI they've committed like $500 billion over, I think, four years.
Like, the first year investment has gone through for sure.
The remaining years seems uncertain.
Uh, I only bring this up because the point is, you know,
we're spending all this money to ultimately update the weight matrix here,
basically. And remember, there are 80 of them.
Like, they could be 80, they could be more than 100.
Like,
we do this, you know, 100 times so there are 100 weight matrices.
We are going to keep updating those weight matrices 100 times.
That's what the data center is for.
So yeah, now let me talk about what are these weight matrices that we are going to
update
by spending billions of dollars on them.
Yeah.
So, this is
the attention block.
So, the most important part is that x is our input. You remember, right?
We started with images here, we're starting with text. So x is just an input.
It has a huge number of examples, and each example has, you know, an array of
numbers.
So, these are the tokens essentially.
We're going, our weight matrices are WQ, WK and WV.
We multiply x with each of these, we'll get, you know, three matrices, QKV.
Then we will do this thing and we will get y, which is our output.
And this is the most important step in a transformer.
That's why I'm going to explain it a lot in detail, because it's the most important.
So, I guess this is straightforward, you understood what's a prediction.
Now we are doing Q multiplied by K transpose.
So K transpose, we added a mask.
A mask
is a matrix that has some values which are hardcoded to zero, and
some values are hardcoded to one.
Actually, no, ma- mask has, a mask has some value zero, some values one.
So when you add a zero, nothing happens.
But when you add a one, this entire thing gets, like, locked into one.
I'll,
I'll explain to you what softmax is, then it will make sense.
So softmax is basically,
if you have a matrix, you want to do softmax on this matrix, it's like, for the first,
you will first do e to the power all the values.
So just all the values that are there, just replace them with e to the power of that
value. So you started with, you know, one, two, three, four.
Okay no, one, two, three, minus one. So e to the power one, e to the power two, e to the
power three, e to the power minus one.
So we just replaced all the values with e to the power of the same value.
Then we will divide each value with the sum of those rows.
So,
instead of one we now have
e to the power one upon e to the, divided by e to the power one plus e to the power two.
Like let's say our values in that row were one and two.
Now that two will become, you know, e to the power two divided by e to the power one plus
e to the power two.
On the next row we had what, three and minus one, right?
So three will become e to the power three divided by e to the power three plus e to the
power minus one.
Minus one will become, you know...
Yeah, maybe I can just give an example here that seems pretty easy to do, I should maybe
just actually give an example here
of softmax.
Softmax...
One, two, three, four, five, six, seven,
eight.This
one...
Equals...
Square.
By equaling square.
B square by B plus
A square.
And cube by...
Cube plus, you know, one by eight.
And cube by...
And cube plus one by, one by eight. (typing sound) Yeah.
(hums) (typing sound) Yeah. So here is an example.
Here, let me just call this an example. (hums) (typing sound) Yeah. Nevermind.
So here's an, here's an example. This is what softmax actually does.
If you want an intuition for what softmax
does,
basically the
large values in a row, they become close to one, the smaller values in a row, they become
close to zero.
If we had done row normalization, that's what would have happened anyway.
Like
if we had just done, you know, two divided by one plus two
(laughs) , basically the large values like two divided by one plus two is two by three.
So that's close to one and one divided by one plus two is, you know,
one by three that's close to zero. But we, because we did this E to the power, it's just
going to be that thing on steroids. So two will become even more close to one and this
one will become even more close to zero.
This three will go even more close to one, and this will go even more close to zero.
Like this thing is almost one basically, and
this thing is almost zero basically.
I mean I,
yeah maybe I'll just compute these values and leave it like why not?
I guess that's 0.268.
Square, 0.731.
0.731.
0.982.
Ah.
(hums)
0.208.
208.
Yeah, so...
Motherfucker.
Yes, so the one was already like a small value went close to zero, two
was big, it went close to one. Like even within the row.
Three went close to one, one minus one went really close to zero.
So softmax does this.
And remember before we do the softmax, we're going to add a mask.
So the mask basically says
some of these values we're just going to add
something so large that, well, only that gets paid attention to or something so
small, you know, that is not going to get paid attention to at all.
Basically, we are going to hard set some of these values to one and some of them to zero,
depending on what the mask says. I'll talk about what the mask says.
But
yeah,
so we're going to do soft mask of this entire thing.
Oh here we're just dividing a constant value.
The dimension of q, so if q is, you know, 1000 dimensions we're just going to do square
root 1000 which is 30, we're just going to divide all the values by 30.
It's not that important. But anyways we did softmax something
and then the softmax thing we are going to multiply with V.
So V is again X at WV.
So softmax something times V.
So this is basically telling us, like in this matrix V, we want to pay attention to some
parts. Like some parts of V are important and some parts of V are not important.
And this entire softmax thing is going to tell us what parts of V are important and what
parts of V are not important.
Oh, and by the way, you should always, always, always remember
this.
Deep learning is alchemy. Deep learning is black box.
We don't have mathematical proofs for anything.When I'm telling you, okay, you know,
here is all the reasons, these are all intuitions. We don't have proof for all this.
There might be some other secret reason why this thing is working well, which we have no
idea about.
Uh, I'm just giving some intuitions of, you know, why does this particular formula happen
to look so good.
You know, this, this thing.
Yeah. So,
this basically says... So,
we are going to
give a lot of weight to some parts of v and we're going to ignore other parts of v.
That's what softmax times means. So now putting this all together.
First, we, what, we multiplied with...
We multiplied Q and K transpose, we added a mask.
Mask said, "Okay, let's hide something." Some things are
as good as zero. We're just not going to consider information.
We are going to softmax it, which means
only some information we pay attention to, some information we don't pay attention to.
That's step two. Step three, we are going to do this on v.
So, at least some parts of v get paid attention to and some parts of v don't get paid
attention to.
So, when you put this all together, it basically says
we are hiding some parts of x, which is our input.
Like, rem- remember, QKV are not some magic things.
QKV are just X multiplied with different weight matrices.
So, we are hiding some part of X from ourselves.
We hide some part of X.
We
pay more attention to another part of X. Remember, this is also X.
Like, it's kind of like, you know, three incarnations of X.
Like...
Yeah, like this is, you know, first incarnation, this is the second incarnation, and the
third incarnation. So the
third... The first two incarnations of X, along with the mask,
or first two incarnations of X with some values masked out tell you which
parts of third incarnation of X to, you know, keep and which to discard.
It's kind of like this.
Yeah. And
then, we get an output.
And now remember, we don't actually know these weight matrices either.
We're going to find them using gradient descent.
We're going to start with some really bad values of WQ, WK and
WV. We are going to compute a loss at the end after, you know,
80 rounds of all this crap.
Uh, we're going to get a loss at the end.
We're going to compute the dL by dW. Like for all these weight matrices, we're going to
get a dL by dWq, dL by dWk, dL by dV. We're going to update
these weight matrices so they're like, are a little bit less bad.
We're going to do this, you know, trillions of times until these are actually pretty
good.
So ultimately, we want to find really good matrices here
that tell us basically in like exactly which situation, what part of X to pay attention
to, what part of X to not pay attention to.
And yeah, remember there are going to be 80 of these.
So,
we are going to do like 80 rounds of this thing with 80 different sets of, you know,
weight matrices.
Sorry, I think that's a lot of things, moving pieces, but yeah.
So, in one round we have
these three weight matrices, which we don't know.
We did this game about paying attention to some parts of our input.
We repeat this with another new set of three matrices with some other in- with, like,
that... The first round's output goes as the second thing's input.
And there we have another three weight matrices. We do again this game.
Then we do it again with another set of weight matrices.
Like this, we have, you know, 80 sets of matrices.
Finally, we get an output right at the end.
Then we compute
the gradients on all these 80 sets of matrices. We update all the 80 sets of matrices.
Then we come back, we redo this.
So, that's how this entire thing is working.
Yeah.
Yeah. Now, here are some intuitions on
why does this technique work in the first place?
Remember,
we don't actually know. This is guesswork. But our guesswork is first of all...
This entire thing can be done parallel.
Like otherwise, if, if you did not know anything about Transformers, you just knew,
uh, this like, okay, look, here is our input, our input is, you know, billions of tokens
for which we know the next token.
And now for some tokens we don't know, we have to predict the next token.
Like, you will typically formulate this as like a sequential problem, you know.
Like, you know, with weather data is kind of sequential or, you know, stock market data
is, you know, sequential. We'll put, you know,
X1, X2, X3, X4, X5, X6. And now, okay, given each of these, you know, pred X7s you'll do
some calculation on X1, you'll do some calculation on X2.
X is the first token, then something on the second token, you'll do some calculation on
the third token. You will go till the Nth token and then you'll try to predict, you know,
what the N plus one token, like...
Most algorithms, if you'll try on this type of problem, are sequential.
So, this could
be, you know, n-grams. Well, n-grams you can kind of parallelize too.
Okay, le- leave aside n-grams. But,
uh...
Where were we? Yeah.
Yeah. So RNNs were a very popular method called recurrent neural networks.
Now nobody uses RNNs, but once upon a time, RNNs were used, and
they do this type of sequential computation.
Like,
you have a weight matrix that does something on the first metoken, then weight matrix
does something on the second token, then something on the third token and so
on.Ah.
And the problem with training an RNN, especially a really big RNN,
is
you have to do,
like you have to update a lot of gradients in sequence.
First of all, that's slower, like GPUs can do computational parallel stuff faster
compared to doing a lot of sequential stuff and then do gradients across all of it.
And there is this vanishing and exploding gradients problem, which is a common problem if
you do a lot of machine learning, which is
sometimes these gradients become very s- close to zero and sometimes they become very
close to infinity.
And when that happens, you need to have some tricks up your sleeve to deal with it.
I mean ideally it should not happen, but if it happens you need to have some tricks to
deal with it.
And,
yeah, one trick
to make sure this happens less is just don't do a lot of sequential steps.
If you do a lot of sequential steps you're more likely to get gradients that go to zero
or go to infinity.
Yeah, so,
that's why
this attention we are doing it is good, because we use this concept of an attention mask
which just says
some parts of this we are going to hide now. Which parts of this we are going to hide?
That actually we can choose. Like we can choose what to put in this attention mask.
Like we can hide the last token and say like predict this.
We can hide the first token and say, you know, go predict this.
We can hide the fourth token and say, you know, now let's predict the fourth token.
And this, when we are training the model we can keep changing the attention mask as we
choose.
But this means now our entire problem is like parallelized, like
we had you know ten tokens, we hid the fifth token.
We know the first four and the last five and now we are predicting the fifth token.
And we will do another gradient descent to predict good values of the fifth token.
So this is parallelized and,
yeah, another good thing about this is we can dynamically decide what to pay attention
to, like
here we have lots of tokens, you know, which of these are important?
Like if I say
my name is Samuel what is
question mark.
Now what is might be more important there and maybe Samuel is not important there.
Like,
if it was my name is Alice, your name,
sorry,
what is question mark, you know, the answer is probably the same, you know, that being
Samuel or Alice makes no difference.
But what is, that is is very important because it says okay here is a verb, now after
this verb you can maybe put a preposition, you can put a noun.
You can't put another verb, that's for sure.
These types of rules
you can reason about. So you need to know which parts of this sentence (coughs) ,
and by the way right now I gave you just six words, right?
In practice they could be in the thousands of words.
Like,
like remember O3s here are giving you thousands of words of output, so it needs to be
able to do this reasoning on thousands of words.
So
it needs to be able to dynamically pay attention to what is important here.
Now some older methods like CNNs, convolutional neural networks were like a really
popular method. Again,
extinct now. Nobody uses CNNs anymore, but
like at most today they, people do like one CNN layer to convert your image into an
embedding and then they fit that into your transformer.
But yeah, nobody trains like end-to-end, you know, CNNs anymore.
(coughs)
So
CNNs basically maintain these sliding windows and like window, window, window, window.
If you have multiple layers you have multiple windows
that decide, okay, now you have to pay attention to this part of the sequence, then we
will pay attention to this, then we will take a smaller window there.
Like if you study CNNs you will understand what I am talking about.
But CNNs kind of fi- hard code this thing what to pay attention to.
LSTMs and RNNs, uh,
maintain one context across the entire thing which is kind of bad.
Uh,
so it's like if you had to do
my name is Samuel what is question mark.
Let us say the context said okay
what is, is paid more attention to and
name is paid more attention to. Cool.
Now, okay we predicted the next token, we said next token is your.
Now again we want to predict, okay, my name is Samuel, what is your
question mark. Now again attention we will say okay name important, what is
important, but actually now
what is is not important. Name is very important because
my name is Samuel, what is your question mark. It's probably going to be name, right?
So we need to be able to now dynamically change, okay, now we don't want to pay attention
to those things, we want to pay attention to a different thing like
we need to be able to change this context like even as we go thr- through each token we
need to be changing our context of what is important to pay attention to.
And LSTMs and RNNs can't do that as well as a transformer can.
So yeah, that's another reason transformers are good.
Yeah. Here is some more, you know, miscellaneous stuff.
(coughs)
In the transformer
I guess tokenization again already briefly talked about
you convert words into numbers. Positional embeddings is basically instead of
saying
my name is Samuel it becomes my one name two is three Samuel four, and
actually we don't store one two three four we
do like cosine one cosine two cosine three cosine four or something which is
yeah a bit unusual but that's the thing again we tried it, it works you know here's
one more in your bag of tricks to
use.The root mean square norm is very
straightforward. It basically says,
like we have done a lot of normalization, right?
We did the normal row normalization, which is divide each cell by
some of all cells in the row. We did soft max which is, you know,
do the same thing but E to the power, you know, E to the power cell divided by some of E
to the power all cells in the row.
So RMS is yet another... RMS is a divide
each cell by sum of squares of all the cells in the row.
This is just yet another way of normalizing.
Yeah, we do a lot of normalizing
partly because of this, you know, vanishing, exploding gradient problem.
Like, we don't want gradients to ever go to zero or infinity, therefore we need to keep
normalizing stuff all the time.
Yeah, and this will ensure our values always stay between zero and one.
Yeah, then there is this thing called residual layer.
Again, maybe I will go back to the big picture.
The big picture is, yeah, we tokenized, we put the positions, we put our images into
something that's not images. So from,
once you have reached here you can forget the fact that they're images.
We just now have
numbers, you know. We have all these vectors coming in, that's it.
So we did, you know, normalizing,
we did our, you know, projection into X, we made into QKV and that
QKV we made back into an output.
We got some output at the end.
Uh, this projection is straightforward, we just...
Yeah, this slide I don't think we need to explain, it's
just whatever output we got from here,
here meaning our soft max times V, we got some output.
We got a Y, we multiplied that by, you know, yet another ma- matrix and we got the
output. So from here we will multiply one more weight matrix, we will get the output.
So here is our multi-headed attention done.
Now we do a thing called add residual.
Now add residual basically means imagine from here we had X, we did multi-headed
attention, we got Y. Now instead of taking Y, we will take Y plus X.
Right, we will take our input, we will add it to the output and we will keep that.
And it is a little bit of a weird trick but it works, we do this in machine learning all
the time.
Again just, it works therefore we do it.
Uh,
yeah, so
Y equals FX was done in the previous step.
So now we will take FX plus X and we will keep this.
And why? Our intuition is basically,
remember we are going to be a- doing this entire thing on like 80 layers, we are going to
do this entire 80 times with 80 sets of eight matrices.
So if you had just done, you know,
F of F of F of F of F of F of F of X like 80 times and then you would do, run
gradient descent on this to find you know the
best weight matrices for each of those Fs.
Like these are all 80 different Fs by the way, like the
weight matrices on, inside each of these Fs are different.
So if you had to do this you are more like again...
Remember sequential is bad for exploding and vanishing gradients.
That is a heuristic.
Uh,
therefore we are more likely to get exploding or vanishing gradients however if
sometimes we
do,
like,
we did F of X plus X, F of this thing plus 3000 thing, F of this thing plus 3000 thing, F
of this thing plus 3000 thing,
and then now we...
And each of these Fs have their own weight matrices.
Now we run unigradian descent across this entire thing
then, uh,
yeah,
then we are more likely to get, uh,
good, uh, weight matrices
at the end. Like we are going to get gradients that are not going to zero or infinity, we
are going to get...
Yeah, I guess you got what I said. Okay, then there is thing called mixture of experts
which I am not describing now. It's kind of a recent trick like two years ago nobody did
this, now everyone is doing it.
And because it's a recent trick also there's a possibility it will not get used in the
future, like
maybe it will and maybe it will not, it's just I don't know how to predict that.
But yeah this is a common trick that has come up in the last couple of years.
Uh, feed forward is pretty straightforward, you can search what feed forward is.
Feed forward typically nowadays is just,
just fully connected network in our two layer. Remember this thing?
Yeah, so this entire thing is like one tiny tiny piece in our entire big, giant big
transformer.
Yeah, so we put here, you know, we put two weight matrices, our W1 and W2 they go in here
then we add a residual again.
We do this entire thing 80 times, we do one last projection step.
Projection just means we multiply it with yet another matrix and we do a ReLU.
Oh yeah, I think I forgot to mention we do a lot of ReLUs like,
like when we do residual we will probably do a ReLU, when we do projection we will do a
ReLU, when we do this projection we will do a ReLU like,
like a ReLU just happens so often it's not kind of worth even mentioning like ReLU is
that common. We just do ReLUs everywhere.
Yeah,
so we have done this lots and lots of computation and then we are going to do
gradient descent,
i- and we are going to find our weight matrices. I think that's basically the overview.
There's one final topic that I need to cover or I want to cover which is the
hyper-parameters which is like
how big should this entire thing be, you know.
How many layers do we pick? How big our weight matrices do we want to pick?And these
are just typical values in the past.
So number of layers is usually between 32 and 128.
How big are weight matrices is, you know, so
GPT-2 was the first good transformer that was trained back in 2019.
It had 1.5 billion
parameters. So parameters is just
how many numbers are in your weight matrix.
So
let's go way, way, way back.
So this
two-layer network we trained, this has,
uh,
so the total number of parameters in this
network are 784 times 800, that's W1, plus,
you know, 800 times
10. Yeah, so we, this is around 600,000 parameters,
635,000 parameter. So this is a 635,000 parameter network.
So GPT-2 is a 1.5 billion parameter network.
There are going to be 1.5 billion numbers inside the network,
and we are going to do gradient descent to find, you know, gradients for each of these
1.5 billion numbers,
and update them a little bit, and update them a little bit more, update them a little bit
more. We'll keep doing that.
So,
yeah,
GPT-2 is 1.5 billion parameters. GPT-3 we made a big jump to 175
billion parameters.
GPT-4 is around two trillion parameters.
You know, our latest models are now,
they've crossed 10 trillion parameters.
Yeah.
Uh, this is a Mixture of Experts thing. I will skip that for now.
So if you're using O3, it's most likely based on GPT-4.
So O3 is, you know, O3 is this thing.
How
to bake chocolate
cake,
no egg.
Yeah. So
this is, uh, O3, this is based on GPT-4.
I mean, it's not just straightforward GPT-4.
We have done some
shenanigans on top at the end with somebody else.
We'll have to talk about in a different lecture.
But
yeah, it's based on GPT-4.
It has two trillion parameters which we found using this gradient descent.
Uh,
each parameter is typically stored as four bytes.
So when we say two trillion parameters, what we actually mean is eight terabytes.
You know, each byte, each parameter is four bytes so 2T into 4 is 8T.
8 terabytes, uh,
nowadays there are also tricks being used, like mixed precision training was used in
DeepSeek, which is instead of four-byte weights, we will use, you know, two-byte weights
somewhere and somewhere we will use four bytes.
Like for some parts of the
network we will do gradients on four-byte weights and some parts of the networks we will
do gradient descent on two-byte weights, and apparently that also works.
So the basic thing is if you do gradient descent on smaller, like use
less bytes for the weights,
you will, you can calculate it faster but also you'll get less accurate answers.
So for training, I mean typically people still use the entire full four bytes.
For inference, inference is, you know, once you've trained the model.
Train means once you've got really good weight matrices, now you just want to do this on
some new answer.
Like you have trained this on our existing data, and on some new data you need to try it
out. When you're trying it out on new data, then people do, you know, two bytes, they
take one byte weight, you know, four-bit weights.
We have, even have now like literally less than two-bit weights.
Like
there's this thing called ternary bits which is like, uh,
you round a number,
uh, I forget it, I'm not explaining ternary bits right now.
But yeah, we can r- make each weight really, really small.
We can make it even like as small as two bits per weight or even less than two bits, so
it's 1.58 bits.
Uh,
so this is our model size.
Yeah, and how did we pick these numbers?
Like who, why did we take the decision of GPT-2 should be this big, GPT-3 should be this
big?
It depends on how much data and how much compute we have.
So data is our input,
like we will train this on some, like initially, you know, we started with 60,000 images.
So here for text we'll start, you know, some few million sentences or some trillion
sentences.
So how much of that do we have and how much compute do we have, which is just how many
GPUs do we have?
And you can calculate the requirements using this law called the Chinchilla Scaling
Law.
Uh, I'm not explaining Chinchilla Scaling Law right now, but you can read it on
Wikipedia, it's pretty straightforward.
Like,
C is how much computation you need, n is the number of parameters,
b is the size of your dataset, and there's like a straightforward relationship between
all of them.
So given some amount of compute, it just tells you, look here's how much.
Like,
given some amount of compute, given some amount of data, here's the loss you're going to
get at the end.
And using those you can reverse calculate, okay, well if you want to optimally use that
compute, you know, how much compute and how much data should be used at once.
You will calculate that. And yeah, an important thing is that usually compute is the
bottleneck, not data.Like usually we have
more data than we actually need
and less GPUs than we actually need
(laughs) . So more GPUs, bigger model.
So you know, this first train on a smaller data center, maybe for an hour.
This maybe was trained in hundreds of hours. Not hundreds, sorry, thousands of hours.
Here, by the time we are reaching here, we are running into you know, millions of GPU
hours.
So
yeah, it's just to train a bigger model, in short, you need more compute and more data,
but
we have data and we don't have compute.
So the more compute you have, the bigger model you will get to train, basically.
And
this is just if you want to train it well.
Like
remember when we were here, when we defined, we never said how many times to do a
gradient update. Like
we could just update the weight matrices, you know, 1000 times and be like, look here's
our performance. It's just this performance won't be that good.
If you want the performance to be really good, you will have to update it, you know,
millions and millions of times. And
then there's a question of, you know, how many GPUs do you have?
How many years can you run these GPUs for and so on.
And yeah, Epoch AI is this nonprofit...
Are they nonprofit or for profit? I'm not sure.
Uh,
let's say it's once in 2028, we'll run out of data.
Like
literally every... all the data on the internet, you know,
will have been fed into a model and there's no more data out there
and then we'll run out of data.
And there are some tricks that might bypass that too.
I- I- My personal guess is some of those tricks will work, but,
uh, yeah, we are going to run out of data in 2028.
And
compute, we're, we're never going to run out of compute.
That's just a question of, you know, like what fraction of the world economy do you want
to invest into, you know, GPUs? And
GPUs keep getting faster. So you know, every 18 months the-
you can get
twice the number of
matrix multiplications per dollar.
So the number of dollars being spent is going up and how many matrix multiplications you
can get per dollar is also going up.
And that seems the trajectory at least for, you know, the next five years, ten years.
So
yeah, you can see more on the forecasts
for how much data and how much compute we will use
in the next, you know, five years here. Like Epoch has forecast.
Ah, my net is fucking slow.
Mm, yeah, so our compute spending is- we're doing 5x every year.
So it's like exponentially going up.
Data runs out in 2028.
We are spending three times as many dollars
on training every year.
So it's first we start with, you know, like
$10 million, then $30 million, $100 million, $30 million.
Sorry, $300 million, a billion dollars.
Right now I think Grok is trained on seven billion, right? Yeah.
So Grok,
trained by Elon Musk, required a data center that cost $7
billion.
Half a billion dollars was spent on Grok itself.
Uh, but just to acquire that ha-hardware, like once you have acquired it, what else are
you going to use it for? It- you're going to just use it either for training or for
inference. So
this is the cost to acquire all that infrastructure, including the GPUs, including the
networking, the cooling and so on.
Uh,
yeah.
I think that's pretty much it for my presentation.
Uh,
yeah here are some rules of thumb. If you don't want to actually sit and solve the
Chinchilla scaling thing directly, here are some rules of thumb that say
for these many parameters, here's how much data you need.
And for these many parameters and this much data, here's how much compute you need.
So yeah, I think that's basically it for my presentation.