[speaker_0] Pre-training scaling 100% does not work or here is an argument
RL scaling 100% does not work. It's just intuitions that
pre-training scale probably doesn't get there, RL scaling probably doesn't
get there
[speaker_1] ... essentially just by scaling RL
just by scaling RL and pre-training, uh, getting to ASI just in the
current paradigm by 2030, uh, in your definition of intelligence
seems very, very unlikely.
[speaker_0] But yeah, researcher, you can right now take a pen and paper,
and come up with, okay, if I had huge amount of GPUs, here
would try, and you can write them down.
[speaker_1] Yeah.
[speaker_0] And then we will ask, okay, well, why haven't these...
Like, did somebody at, you know, OpenAI, Anthropic try them and they failed, or did
nobody try them?
[speaker_1] Uh, maybe you're right and we're on the vertical part of the S
I think if you just plot it from AlexNet, it's mostly the same
paradigm. Uh, and I would count AlexNet
as one breakthrough, Transformer as another breakthrough,
breakthrough.
[speaker_0] Hello, everyone. Uh, my name is Samuel Shadrach.
I graduated from IIT Delhi. I've been following the whole
AI timelines debate for s- a while now.
By when will humanity get super intelligence?
Is this a good thing or a bad thing? What can we...
And now I have a fairly strong opinion it's a bad thing. We should stop it.
That's a whole separate discussion that we can have elsewhere.
Today, I have with me, uh, Raghav. Uh, we are
going to specifically just discuss, uh, AI timelines.
Uh, when do we think, you know, super intelligence will come?
Uh, you know, assuming the research, current development continues.
Raghav, would you like to introduce yourself?
[speaker_1] I'm been a friend of Sam. I've been following the AI safety debate
for the last five, six years now, uh, and very
interested in the subject. I have some strong opinions as well,
as Samuel, but, uh, I have slightly differing opinions from
That's why I'm keen to talk about this.
[speaker_0] Yeah. First of all is, yeah, uh, the whole pre-training
scaling, which right now everyone says is dead.
Even now I'm not yet convinced pre-training scaling is dead.
Yeah, I think there is still a small chance
and you just extrapolate the pre-training, whatever chinchilla,
whichever scaling curve that has been running for the past few
we do that a few more years with new GPUs coming.
I still think there is at least a little bit chance
ASI, which I'm claiming. Uh-
[speaker_1] Okay
[speaker_0] ... a lot more realistic to me
uh, some way of scaling up RL. Now,
that may not be, like,
just again, you just do the current, you know, RL scaling thing
compute. Maybe one breakthrough is required there.
I have, like, more probability mass than, okay, we need at least one breakthrough
on how to do the RL thing better or some other breakthrough.
I have a little bit less probability mass than, okay, you just blindly scale up RL
and it works. But yeah, that's the thing is, like, we don't know.
I have not h- seen like, okay, here is an argument that tells me that
pre-training scaling 100% does not work, or here is an argument that tells me
RL scaling 100% does not work. It's just intuitions that
pre-training scale probably doesn't get there, RL scaling probably doesn't
get there. We probably need another breakthrough,
really knows. Uh, that's broadly the path,
uh, of, you know, what technical capabilities will be
Uh, then there is, okay, the most probably we'll need one more
breakthrough. Why do I think, okay, even if we need one more breakthrough, we
will... There's a good chance we get it in the
For that, you have to extrapolate, well, in the last 10 years, how
many major breakthroughs have happened in machine learning, deep learning, and,
well, actually three or, like, three or four major breakthroughs have happened.
So based on that extrapolate, okay, it doesn't seem surprising to me
if in the next four years another, you know, smart researcher figures out yet
another breakthrough. So there is that kind of thing.
Then I have some specific heuristics and thing.
If I were an AI capability researcher, if I had a huge number of
GPUs, what are, you know, my crazy research hypothesis I might
try for how to speed up RL? And to be clear, I think this is
extremely dangerous thing to do. I think this is bad for the world to do.
But, you know, if I wanted to do this, like, I think, okay, there are some
hypothesis you could try. Uh, what else?
Yeah, this is all, like, very specific to AI research capabilities,
trajectory. And then I have, like, more very high level big picture kind of
Like, you know, why is super intelligence important?
Uh, you know, why the very fact that GPT-2
exists should update your entire models of how the world works.
The fact that
bunch of matrix multiplication can do, you know, like, uh, like
speaking human language rather than animal language, why this is a big deal?
Like, you know, mat- why did matrix multiplication beat the
literal billions of years of evolution that goes between, you know,
uh-
[speaker_1] I'm sorry, I, this, I don't think I'm,
only. I mean, like, maybe we've not discussed this earlier.
[speaker_0] Yeah.
[speaker_1] But maybe we can get to it.
[speaker_0] Yeah, sure.
[speaker_1] The other one I know you're just saying, but this one seems new to me.
Maybe you're framing it differently or something. Yeah. It's fine.
We, we'll probably get to it in the order, but I'm just flagging it that I probably
need you to double-click on this instinct.
[speaker_0] Sure. Uh, I, I just mean like, uh, human language has features that
are not present in animal language.
There are some linguists that have studied that,
like new evolutionary adaptation compared to most animals who don't really even
use language. They kind of just use sounds
And all of this has taken, like, literal billions of years in evolutionary
history to build, and on the other side you have a bunch of
bunch of GPUs in 50 whatever years of AI research history, and
they have been able to crack human language.
Like, that itself tells me, "All right, so intelligence is
likely easier to build than I thought." So that is like-Yeah.
So even just looking at GPT-2 tells me like, oh, okay, so maybe now the
singularity could happen in my lifetime.
Like, you know, until now I was thinking this is some extremely far
Now it looks like, okay, this could actually happen. Uh, what else?
Yeah, I mean, and I have outside level, outside view kind of stuff like,
okay, which experts are actually correctly predicting this?
Which experts are badly predicting this?
I think a lot of people have been consistently badly predicting,
uh, AI trajectory. Yeah, like the people who
their predictions are coming more correct, and the people who keep making the
pessimistic predictions, their predictions keep coming wrong.
So there's that kind of stuff. I think that could summarize my whole
[speaker_1] Fair enough. We'll start taking it one after the other.
Uh, cool. So starting with, uh, you said pre-training, scaling,
Pre-training scaling is work, and pre-training scaling is not dead.
I think when people say pre-training scaling is dead, they don't mean
more parameters and adding more data and adding more compute doesn't lead to
better loss functions that leads to more capabilities.
I don't think anybody denies that.
Uh, people are uncertain that, uh, people are
say-saying that it could stop, but there is no evidence to believe
stop, and I don't think any serious researcher
The argument against pre-training mostly comes from the fact that, um,
it is economically unfeasible to scale because the, uh, amount
of
resources required to do the scaling gives you log.
It does not linearly increase, it gives you log of the intelligence.
Uh, so like increasing, uh, your compute by ten x et cetera gives you
the amount of intelligence or capabilities and double the amount of
goes.
[speaker_0] Exactly. Double the amount of loss,
loss number leads to this, uh, capability.
[speaker_1] Sure. And so which also means that, uh, so th-then the
debate essentially shifts to not that pre-training is dead,
many ten x's can we do and how many doubling of intelligence or,
slightly higher intelligence will essentially do it.
Uh, I think you'd mentioned this in the OOTD
GPT 4.5 was a big update for me, saying that if you just go
on cranking pre-training, you might get a nicer model with not a
capabilities. There was nothing 4.5 could do that was so
far ahead of 4, uh, that essentially I think that
this would essentially
defeat, uh, like th-th-there would be new
Uh, o1 was in a different update because o1
And essentially, this is in the pa-current paradigm.
So my submission for the pre-training argument is that not that pre-training is
dead in the sense that you can't technically, uh,
burn all, all the GDP in the world and like get a
few doublings, etc., out of it. Uh, whether essentially A, there
is a-- it's economically feasible to do it and B,
even if it was economically feasible to do it, uh, is, uh, like
essentially maybe the returns of on, on, on it that again is probably
not worth it. Maybe spending hundred x the amount of
times better model, uh, probably starts breaking a lot of other
Uh, having said that, we are also almost on the edge
of how much compute we can build. Everything is stretched out to its limit,
uh, especially by twenty-thirty.
The year you said, uh, fabs are already built
out, etc. We have nowhere close to, uh, let's say
four x-- four, four increasing our, uh, pre-training.
We probably can do one or two more scale-ups in the next two
Sorry, in, in the next four years. Uh, and that's maximum that we can do.
Uh, even like we do not have enough fabs, we do not have enough electricity, we do
not have enough... Some insane amount of physical bottlenecks
the amount of compute in the world.
Um, currently we are doing, uh, I think forty or
fifty gigawatts of...
Fifty gigawatts? No, thirty gigawatts of, uh, compute capacity is
what twenty twenty-seven will get us.
And, uh, if everything goes according to plan
like all the supply chains get stretched exactly to the right limit,
our best bet is to get to one fifty to two hundred gigawatts a year, which is again
like six times more compute, not ten x more compute, um, from what
we have right now. And like that's two hundred gigawatts per
Uh, and that's assuming all that compute goes into training
etc. So, so there are like lots of physical limitations
till twenty-thirty that do not allow for arbitrarily amount
Uh, we've already kind of pressed to the
Uh, buying a laptop costs like three hundred,
Costs three hundred, four hundred more.
I don't think that the world can essentially take four orders of magnitude
compute scaling that readily now. I think we,
Coming to RL, I agree that essentially RL scaling
There are two points that I want to make about RL.
RL scaling, uh, is also extremely expensive.
Uh, it's more expensive than pre-training
Uh, the other point that I want to make about RL is, uh,
gives us general capability increases, RL gets us very
jagged increases in capability. You only get increase in capability in
domains like coding and math, uh, which essentially kind of
defeats the, the specific model of intelligence
that it will be better at everything by twenty-thirty.
So if we, if we cannot find good RL candidates
loops for different kinds of event, we might not even solve, uh, forget
solving for like robotics and like other things.
Even in just like atoms world, we might not be able to solve all of it
to human level because we will not just find enough RL data, etc.,
or enough closed loops, etc., uh, uh, to essentially do it.
Uh, so, so I don't think that RL also scales
arbitrarily that you can just thousand x the RL compute and actually get
away with it. Uh, there is a limit to how much RL compute you can
RL also just gets you increases in capability.
Yes, economically viable, uh, the economic, the killer use
case for LMS currently is coding, and coding is economically viable, etc.
There's a small, uh, probability gap I have.
This leads to some sort of recursive self-improvement or some sort of, uh,
breakthrough, etc., that happens because of this.
Uh, but aside from that gap, uh, which sure, we can talk about
Uh, but aside from that gap, uh, the path essentially just by scaling
RL or just by scaling pre-training or just by scaling RL and pre-training,
uh, getting to ASI just in the current
uh, in your definition of intelligence seems
unlikely.Um, to me, like forget 25%, I would give
sub 1% chance in the current paradigm, uh, in these specific
circumstances. Not to say that, uh, I have a much higher probability was
might get to superintelligence by 2030, but, uh, that's
essentially assuming technology that haven't been invented yet come into
being. Uh, so that was the second point about RL.
Uh, your third point was tied into this, is like, okay, we might need one more
breakthrough. Uh, current RL might not...
You think there's a chance that might not be enough.
Maybe it's enough, but we might, maybe it's not enough.
Maybe we need one more breakthrough.
And, uh, uh, your submission there was that, hey,
breakthrough will happen, uh, because look at the last 10 years,
we've gotten transformers and we've gotten pre-training scaling,
RL. So looks like breakthroughs are coming very, very quickly.
Um, I think is, uh, this, I don't think that,
uh, there is enough data points for you to
Uh, even sta- in the start of GBD, despite essentially almost
all of world's intelligence and attention going to this problem,
RL scaling, which we've cracked, uh, there's not been another scaling
paradigm that we've cracked. So apart from pre-training and RL scaling,
people are ready to throw computes at other scaling things, and that's not
something that we've cracked. I'll take your rebuttal.
[speaker_0] Oh, rebuttal.
[speaker_1] Like what other scaling paradigm have,
RL-
[speaker_0] No, no.
[speaker_1] Despite throwing insane-
[speaker_0] Okay, it has a lot of attention,
You're saying this mean that other breakthroughs-
[speaker_1] No, no other breakthroughs. I'm saying other breakthroughs...
I'm saying, uh, other breakthroughs are possible in the sense they're not
physically impossible. But A, do you disagree that there are other,
things that we can just throw,
s- other, other scalable things that we can throw compute money,
more intelligence aside from, uh, like just regular pre-training
and RL?
[speaker_0] I think there is a huge backlog of research hypothesis
at all these AI companies.
[speaker_1] I doubt it.
[speaker_0] Like lot of very-
[speaker_1] I doubt it
[speaker_0] ... obvious things to try, but because compute is scarce,
decide-
[speaker_1] I don't know
[speaker_0] ... okay, which ones to, uh-
[speaker_1] I get it, but essentially, I, I, I, I, I hear that argument saying
that obviously we can improve our models in X sector, but if there were such such
obvious scaling paradigms that essentially could have been done
than the current paradigms, uh, I think you would see some evidence of it.
You would see there's, there's enough time essentially going into
that-
[speaker_0] No, I, I hear all excuses
[speaker_1] ... that-
[speaker_0] I'm saying like you right now, you are not an expert AI researcher.
You can right now take a pen and paper, sit for half a day and come up with, okay,
if I had huge amount of GPUs, here are 10 hypothesis I would try, and
you can write them down.
[speaker_1] Yeah.
[speaker_0] And then we will ask, okay, well, why haven't these, like,
know, Open-- Anthropic try them and they failed, or did nobody try them?
[speaker_1] I-
[speaker_0] If nobody tried-
[speaker_1] ... doubt it's that easy. Can you name,
that hasn't been tried that you think has a higher chance of it
an RL level breakthrough if tried?
[speaker_0] I'm not saying any one idea if I try that will definitely work.
I'm saying-
[speaker_1] No, any example of an idea
If nobody's tried it, I didn't find any research papers on it,
we'll probably get ASI.
[speaker_0] Uh, if you just want random ideas,
Uh-
[speaker_1] Sure.
Just to understand, like, what kind of ideas do you think
these guys are so compute constrained that if there is an RL level breakthrough
just sitting on like some researcher's notepad and they've not had the time.
Because even for RL, they didn't actually have to use
the idea. Uh, the idea, the idea, the, the-
[speaker_0] Yeah. Uh, one example, most of the training still happens in very
code, like, you know, PyTorch, you know, like four-bit, eight-bit floating point
numbers. If you really wanted, you could optimize all this way down.
You could literally run training inside an ASIC. You could optimize the code.
[speaker_1] Yeah. So, so for example, that's like a very bad idea. No, no, no.
So for example, running ASIC, so, so that essentially assumes
like a lot of other things need to move in the world.
One, the amount of GPU capacity that already exists allocated in the world
needs to go away. Secondly, there's a reason why even of co...
Like doing this does not get you that much compute because essentially
compute constrained, you're memory constrained, uh,
training and you're basically memory bandwidth constrained, not even memory
constrained. And, uh, yes, sure, like we have like silicon
photonics, and we have other breakthroughs
engineering problem. Uh, I don't think it's as easy as like, hey,
on ASIC and we get like 100X speed up and nobody's had the time
Uh, there's so much incentive for any smart
If you think it's that easy to go replace the entire compute
ASIC and people have just like not tried it because of some other
crunch, I, I think you're mistaken.
[speaker_0] No, no, I-
[speaker_1] Like there's insane amount of like capitalist in-incentive to do it.
Like, th-which would be like, "Hey." I'm saying if there's any other
uh, the amount of compute you need to quote
small. Uh, and therefore if, if there were these insane...
o1, for example, didn't require too much compute on forward
could work. Uh, if you had such an idea, you could show that, hey, this
works and we want to use this to scale our models and it's sitting in the labs and
we just think, but looks like there's n-none,
[speaker_0] Uh, oh, okay. Yeah. I'm saying there are a lot of research
require a lot of compute to prove even as a proof of concept, okay, this is worth
exploring.
[speaker_1] I, I doubt it.
[speaker_0] Yeah.
[speaker_1] I think they all-
[speaker_0] The idea is-
[speaker_1] They all show signs of life. All of these research ideas show signs of
life much before you have to scale them to get gains out of it.
There are very few research ideas that quote unquote "only get unlocked at,"
if you only train them for, like only if you spend billion dollars of compute is
the first sign of life you get from that research idea, uh,
research idea saying that, "Hey, if I just keep throwing compute at it"-
[speaker_0] In research, almost every breakthrough
Like only after you scaled it to literal billions of dollars you saw that it was
working.
[speaker_1] No. Give me one research idea that's gonna
GPT-1 was like a very small model.
[speaker_0] RL or-
[speaker_1] RNN was a very small model.
[speaker_0] Uh, GPT-
[speaker_1] RL, for example, didn't require billions of dollars.
So o1, for example, was a very cheap model.
Once GPT-4 was trained, training on chain of,
of thought and doing RL was like a very, very small experimentSo,
uh-
[speaker_0] Can, uh, do you have numbers on that?
[speaker_1] Yeah, yeah. So, so there's, A, there's ...
I, I'll find out where I read about this, but basically the first version of the O1
model was just get tried in a lab, et cetera.
For example, uh, you can take Llama 3 right now,
and, uh, you can take-
[speaker_0] No, no, not, not about right now.
Back then when O1 was first tried, like my guess without having-
[speaker_1] Like I'm saying, you can make it more efficient.
[speaker_0] S- sorry.
[speaker_1] You can make it more
efficient. Yeah.
[speaker_0] Yeah, like without having read it, my guess
required at least $10 million on top of the
GPT-4 training cost, and like I have not actually checked the
but yeah.
[speaker_1] You needed the GPT-4 to get trained first,
and then you found a new scaling paradigm on that scaling paradigm.
[speaker_0] Yeah, yeah.
[speaker_1] That I agree.
[speaker_0] I'm saying first you had the whole-
[speaker_1] But go from four to-
[speaker_0] ... GPT-4 training cost, then GPT-4 trained.
[speaker_1] Right.
[speaker_0] Then you had-
[speaker_1] But O1, when you look at-
[speaker_0] ... to research hypothesis. Each of those hypothesis took at least $10
to test, and 10 million is a random number.
[speaker_1] No.
[speaker_0] I think it's actually important. Uh-
[speaker_1] I don't think it takes $10 million of test.
I think it takes like a few hundred thousand dollars,
dollars to test a research idea to see if that it has any
sign or any chance. Yes, like you might see like different gains
losses, et cetera.
[speaker_0] That is then possibly 100 million is what I'm claiming.
[speaker_1] I doubt it. I don't know if any of all these ideas take 10 million.
For example, O1 didn't take 10 million post GPT-4
[speaker_0] We can actually go and check that maybe.
Like I know this is-
[speaker_1] No, no
[speaker_0] ... not public information, but we can go and see like-
[speaker_1] No, no, I think we can check because the amount of
Yeah, sure, because the actual amount ...
So the idea, the way research works is get an idea,
this thing. If y- if you see any sort on it or anything
"Cool. You know what? This warrants more investigation,
actually go improve something in the form." And then you can do optimizations
it, and then you can make it better, et cetera.
But the first amount of like this thing, the sign of life saying that this has,
this is probably has some legs can come from not that much money.
[speaker_0] Yeah. This is-
[speaker_1] And then obviously getting actual real-
[speaker_0] ... GPT works, however, ML is not like this.
[laughs] I think that's my actual method.
[speaker_1] All of ML has been like this. Like pre-training, for example,
was a scary small model. GPT-2 was a very small model.
Uh, it took like, uh, probably 150 or 2, like the, the initial
GPT cost like a million dollars, I think. Not that much.
Uh, GPT-3 again, like took like slightly larger amount of
but nothing compared to right now.
And the amount of, uh, the only reason you go from GPT-2 to 3 to
4 to 5 or whatever subsequent models is because you see gains
from scaling all the time. Uh, RNN, for example, you can see that, okay,
scale the RNN paradigm, you get gains from it.
Uh, and then, then you stop seeing gains, so that's why RNN didn't scale or
whatever. Or, uh, then they went to LSTM.
LSTM didn't scale, and then they went to transformer.
Transformer scaled fairly much, and like, cool, we found a scalable paradigm.
The idea that you can see gains and then you try to scale it
So-
[speaker_0] Yeah
[speaker_1] ... uh, so it's not that, oh, I have to like literally roll dice of $10
each time to get one idea. It's not that random.
[speaker_0] Okay, I'll make a tighter claim. After GPT-2
had come out, if you wanted to try out any ML research
hypothesis, you probably needed at least $10 million
life or not. Like for, for most of the research hypothesis-
[speaker_1] No, you could use the, uh-
[speaker_0] ... you needed at least $10 million since GPT-
[speaker_1] No, no. So the id- the, the idea is that you train GPT-1,
saw that there were gains on it. The obvious thing to do is that, "Hey,
found a scalable paradigm. I think I can scale it further to see
gains."
[speaker_0] Sure.
[speaker_1] And then you train GPT-3, and there are still gains from it,
res- dead research ideas that might show signs of life, but o-
it, they start breaking.
[speaker_0] Yeah, I'm saying to try any other idea,
pre-training scaling. If you have any other idea apart from
GPT-2 came out 2018, right? So after 2018, if you had any other
idea besides the whole pre-training scaling thing, you wanted to test it out,
would need at least $10 million for most ideas to even test out and see if this
has any life or not.
[speaker_1] And that's also trivial amounts of money if...
I think if people who know about how these things work, it's not completely
unintuitive. They have, they have a big idea of test.
I, I'm not saying is that, uh, there might be a breakthrough that
in some old research paper or in scribbled in all the diaries
overlooked, uh, but it is not as low-lying a fruit as you
if, if only we had just spent more compute,
thousands of thousands of scalable paradigms." Finding scalable paradigms is
really, really hard. We've only done like two ti-
last 10 years and two times in the last 50 years and two times in the last thousand
years, and, uh, essentially, uh, uh, just because we've gotten
lucky, just because we got lucky doesn't mean
happening in the next two, three, four, five years.
It's just like there'll just be like scalable paradigms after scalable
that will just keep showing up because it's shown up last two times.
[speaker_0] Okay, so we have identified some disagreement
Uh, how do you think we can resolve this?
Like what data points will work or what arguments about how do you think
[speaker_1] One data point that would work, one data point
move you is that if you look other, uh, if you look at other fields, you look
at biology, et cetera, uh, just because like there's one breakthrough
that essentially leads to a different class of
discoveries or drug discovery happening.
For example, you get, uh, let's say
discovery through, like, for example, RNA delivery of drugs,
uh, like the mRNA vaccine, et cetera, which you can like modify RNA and you can
inject in people, et cetera. That means that, yes, a lot of, a lot
of, uh...
That, that was a big deal, like get a new branch of medicine,
doesn't automatically mean that the amount of new breakthroughs will
increase. If anything, what we've seen is that there is actually a slowdown in
amount of new ideas and new researches in every mature field
more, more attention goes into it because all
Uh, so finding the next breakthrough is not a linear process.
It's actually a super linear process. Not like it's a log process.
You have to like spend 10X, 100X more resources to get more ideas,
and, uh, you pluck the low-hanging fruits very,
is not that hard. It's not that easy.
So-Um, so other field at least do it.
You can argue that M-ML is different for some reason.
I don't know why scientific idea is, like, inherently be different in, uh, ML
because like, oh, ML is different.
Because mostly what happens is if enough eyeballs look at a problem, uh,
they look at all the low-hanging fruits, then they go to the second level of
low-hanging fruit, and they keep doing this, and they,
hypothesis. And like, okay, cool.
Uh, this has been already done in physics, for example, uh, or
like chemistry, et cetera. We don't expect like crazy amount of math
come out, uh, by a mathematician, uh, or like a
great, like, new, new lines of math schools,
Uh, and I feel like with more attention on a
It becomes harder to e-curve. S-curve is not easier.
[speaker_0] Okay, uh-
[speaker_1] It depends on what part of the S-curve you're on, I think.
[speaker_0] Yeah, I think that S-curve analogy is good.
So yeah, if we are comparing to other scientific
field in which experiments are expensive.
So, like, we should not compare this to, like, theoretical math where you just need
person sitting with pen and paper.
Like, something like drug discovery is a better analogy for this.
And by the way, I do think ML is a bit different,
analogizing with other fields, uh, in drug discovery.
[speaker_1] Or like theoretical physics, for example.
[speaker_0] Sorry, y-you mean theoretical physics is cheap
[speaker_1] Is also expensive. It was cheap at some point,
like, breakthrough with pen and paper.
But now if you want to, like, uh, experimental physics, sorry,
physics-
[speaker_0] Mm. Yeah
[speaker_1] ... uh, you could make like a... Yeah,
to do any new physics.
[speaker_0] Sure. Yeah. Okay, fine. Experimental physics would work.
Uh, yeah. Now to actually argue about experimental
physics or drug discovery, I will have to actually read more about
physics or drug discovery. [chuckles]
[speaker_1] Uh, but do you agree that, that, like-
[speaker_0] Also, we have to-
[speaker_1] With the-
[speaker_0] Yeah, no
[speaker_1] ... enhanced attention and-
[speaker_0] Also, we have to pick a time period. Sorry.
Uh, also, like, if we take, you know, drug discovery or experimental physics as
example, we have to take a time period in the
was known that, okay, this thing is, you know, in like boom phase and like, you
know, lots of new capabilities are coming out.
Like, like in ML right now, we know we are in that sort of phase.
Like, there may be new things we could try.
Like, it's not like, okay, it's like a dead field-
[speaker_1] Sure
[speaker_0] ... mature field.
We have not reached-
[speaker_1] Sure, sure. Yeah, I agree. And like example,
that. I don't know about experimental physics,
also, where a bunch of like new physics
1920s. There was a activity that came out.
There's like, like, a bunch of these new...
All of them essentially got started in 1920s.
There was one period in the 1600s that happened.
Uh, but, uh, I agree that we might-
[speaker_0] Studying that particular time period to study, you know,
that happened there? And after a few breakthroughs-
[speaker_1] Yeah
[speaker_0] ... came out, now to extrapolate, okay,
breakthroughs come out, and how expensive will it be to run-
[speaker_1] Sure, sure
[speaker_0] ... experiments? I think that's the kind of-
[speaker_1] And I agree
[speaker_0] ... study.
[speaker_1] And I agree. And, and, and then essentially, right,
part of the S-curve, that then there, then you should
expect more breakthroughs to come out.
If you're on the horizontal part of the S-curve, then you should think, then you
should expect less discoveries to come out.
Which part of the S-curve you are on, I don't think either of us know.
Uh, but I think-
[speaker_0] I'm claiming-
[speaker_1] ... the more time-
[speaker_0] That's my claim. And also, sure, some decent probability they're not,
[speaker_1] Okay. Uh, what will be your evidence to saying
Like, how are you so sure? I have zero base to say that this thing,
can, on hindsight be like, "Oh, looks like we were on the vertical part
[speaker_0] Okay. And for me, it's just extrapolate last five to 10
data points. Okay, which year did, uh...
Well, actually you can go back to , you know, which year did AlexNet come out?
[speaker_1] Uh-
[speaker_0] Then which year did, you know, transformer come out?
Which year did-
[speaker_1] That-
[speaker_0] ... GPT-2 come out? And just put these on like a year
versus, you know-
[speaker_1] That is-
[speaker_0] ... new breakthrough kind of graph,
this look like your S-curve is saturated?
Yes or no?" And no, it doesn't look like there's-
[speaker_1] And this is like an outside perspective.
You don't have to trust it, but Ilya Sutskever, who
was responsible for GPT, was responsible for RL, uh, comes
and says essentially all the good ideas are down and we need to spend some time
doing new research. And this is the time for...
I don't know if you saw that episode or The R Kesh, but he's like,
scaling is over. Now is the time for new research."
[speaker_0] What are Ilya Sutskever's timelines?
Are they less bullish than me when I'm saying 25% ASI 2030?
Like, does Ilya have like less bullish timelines
[speaker_1] I don't know, but I think that's a relevant.
[speaker_0] Sorry?
[speaker_1] I think that's irrelevant. I feel-
[speaker_0] No, no, you brought up Ilya-
[speaker_1] Yeah. No
[speaker_0] ... then I was like, okay, like does Ilya agree with me already,
thing.
[speaker_1] It doesn't matter if Ilya agrees with you.
What Ilya does agree, disagree with you on is
that the low-hanging scaling fruits have been plucked and we need to go find new
scaling breakthroughs, uh, which he's confident that he will find,
economic incentive to say that. Uh, but he's saying that, "Okay,
are down, and now we need to find something new to
[speaker_0] Okay. No, but if you strongly defer to Ilya on
this question, then Ilya's actual timelines are-
[speaker_1] No, I don't. I don't,
I don't, I don't defer strongly to Ilya on this
All I'm saying is there's one extra evidence point saying that the guy who was
involved with all these three breakthroughs comes and says
no low-hanging fruits anymore, and we have to go find more, uh,
should update you that we are probably not on the vertical part of
[speaker_0] No, no, no. Uh, I took a different lesson from this.
Uh, like Ilya has, again, as bullish as me timelines,
low-hanging fruit is picked. What he means is, okay, the
low-hanging fruit is not like one month low-hanging fruit.
It's like two years, three years low-hanging fruit.
[speaker_1] No, I think his time, his timelines are significantly longer.
I think he's like in the next eight to 10 years will probably be some areas
of research that we need to find to scale, and yeah.
[speaker_0] Okay.
[speaker_1] I, I think his is probably longer.
[speaker_0] So, so again, I'm like, yeah, if you want to debate specifically Ilya's
I think actually we have to go and find his timeline.
[speaker_1] I don't want to debate Ilya's worldview.
I'm saying that this, the, it's not a question of whether Ilya
The question of, uh, Ilya has an evidence point
may or may not update you on which part of the S-curve we are on.
[speaker_0] Yeah. So for that, I need to first even understand does Ilya
or does he have some major disagreement?
[speaker_1] No, he doesn't. He doesn't. He, his timelines are, uh,
[speaker_0] So then want to know what his timelines are.
[speaker_1] I think he said the next... I might have to check on this, but
Darkish episode, he says that like six to eight years, uh,
and then we'll find something that we can scale.
And, uh, there are no good scaling candidates in
[speaker_0] Yeah, no, to continue this, like, I think, like,
more... Like, are you sure about this? You know.
[speaker_1] Yeah.
[speaker_0] Give me more context and-
[speaker_1] No, no, I'm saying, I'm saying, I'm saying that,
S-curve we are on, uh, there's uncertainty on it.
Uh, maybe you're right and we are on the vertical part of the S-curve.
I think if you just plot it from AlexNet, it's mostly the same
paradigm. Uh, and I would count AlexNet
as one breakthrough, Transformers as another breakthrough,
breakthrough.
But apart from that,
I don't see why-
[speaker_0] Yes, once, uh, no, but there are also like minor ones
[speaker_1] ... like, I don't know any other paradigm that
Okay, I won't even count AlexNet 'cause AlexNet
works. Uh, I would just count Transformers
that, uh, show that scale is all you need.
Uh, and then there's n- there's not been a third candidate for
need, or a third S-curve that you can stack on top of these things.
Pre-training was one S-curve. We exhausted it.
[speaker_0] There are multiple points. One is like proof that Transformers
all, and then there's a second data point
Or even with, like-
[speaker_1] Transformers are useful at all is...
Transformers are only useful because they can scale.
[speaker_0] No. Uh, GPT-2 was-
[speaker_1] Because-
[speaker_0] ... useful. Like, it was a breakthrough by itself, even
anything about whether GPT-2 would scale or not.
[speaker_1] If it doesn't, then... No, what I'm talking about is that if you
assume that scaling is what gets you smarter models, uh,
subscribe to that worldview, then you need paradigms that can scale.
Pre-training is one paradigm that can scale.
[speaker_0] Right now. At the time GPT-2 was invented,
ML research community didn't believe it.
[speaker_1] Sure. And I, like, I'm saying that it's, that's, that's
irrespective, like, that's immaterial.
What I'm saying right now is if you believe that intelligence comes
scaling things up-
[speaker_0] Sure
[speaker_1] ... then scaling a paradigm up,
scaling up.
[speaker_0] Yeah.
[speaker_1] In doing so, uh, we have found only two paradigms that have
scaled up.
[speaker_0] We have found only two paradigms that have scaled up.
No, I mean, why does, uh, the whole scaling fully connected
networks back in twenty twelve, twenty thirteen, why does that not count?
Why does scaling LSTM not count? I, I'm not clear.
[speaker_1] Because LSTM didn't scale. CNNs didn't scale.
[speaker_0] What do you mean? CNNs do scale.
[speaker_1] As in, like, you get diminishing returns from scaling
CNNs, for example, cannot-
[speaker_0] Yeah.
[speaker_1] People try to do language experiments on CNN,
People try to do language experiments on LSTM, they perform to a certain level, but
essentially then they start degrading.
So th- these, these paradigms have lots of
limitations on how much they can scale.
[speaker_0] Sure. So they, they did scale up to some amount,
saturated, I guess. Same. Like you had some estimate, I guess.
[speaker_1] So then, then it's a failed candidate. Yeah,
Like, like a good candidate is that, hey, it doesn't matter how much compute
we throw at it, just keep scaling.
[speaker_0] No, no, it means that back then it succeeded, right?
CNNs, we did scale up to some amount, then we realized we're getting diminishing
returns.
[speaker_1] No, but that's what I'm saying, like,
that way, like... No, no, that didn't...
I don't know what your definition of success is,
to superintelligence. Candidates that can get us to superintelligence are,
are candidates that you can throw.
There is no limit to how much, uh, or there's no visible limit to
how much compute you can throw at it.
For example, if tomorrow we find out that pre-training has stopped scaling,
call pre-training a failed candidate.
Currently, we have two candidates that can absorb insane amounts of compute
et cetera, and can keep expecting gains from it.
[speaker_0] No, even the paradigm that gets us to superintelligence might
saturate somewhere. E-e-every candidate-
[speaker_1] Sure
[speaker_0] ... can saturate.
[speaker_1] They would, can saturate, but I'm saying...
A-and at that point, essentially, uh, you can either draw a line and say
intelligence is good enough. Assuming that, okay, so you're in a,
where you do not have enough, uh...
You've not achieved the level of, or like, not achieved, that you don't want to
achieve. But I'm saying like, uh, you think that you can still throw more
at this and get more intelligence out of this.
Uh, and, uh, in that paradigm, there are only two things that can
absorb seemingly infinite amounts of compute.
Uh, and there's not been a third.
[speaker_0] I have a problem with your infinite thing.
It's-
[speaker_1] Seemingly infinite in the sense that, sure,
how much you can do it, or there might be like some...
There are obviously practical limits to it, but there might be a theoretical limit
as well. And I'm saying that maybe before we get there, uh,
superintelligence comes and then you, you've achieved your goal
to scale further.
But until now, there's no evidence to see that there
[speaker_0] Sorry, I'm still not super clear what your claim
Can you like summarize this entire argument?
[speaker_1] Maybe I'll, maybe I'll try to explain it in a
A successful paradigm in my book is one that can
absorb all the compute capacity that you can reasonably throw at it at
point in time.
[speaker_0] That point in time means what? In that year, how many GPUs had
humanity manufactured?
[speaker_1] So it's also... Yeah, in that point in time,
So, so in twenty twenty-six, uh, there are two successful paradigms that can
absorb all the compute and still keep on giving gain.
We have not exhausted these two paradigms.
[speaker_0] Right.
[speaker_1] And we still have like returns to get from there,
essentially get us to smarter models. There are just two of them. And twenty...
And so I'm saying like you keep extrapolating it, our, our, our lim-- our,
thing. So in twenty twenty-- by the time we get to twenty thirty,
that these two paradigms will not get to superintelligence because there are
practical limits to scaling them, if not theoretical limits.
Uh, and, uh, we need either a third paradigm
seventh paradigm to actually keep stacking these S-curves to get us
to the world that you are claiming we'll be in, say,
[speaker_0] Uh, okay. Sure. My probability
ofPre-training scaling plus RL scaling gets us
to ASI by 2030 is less, let's say, less than
10%.
[speaker_1] Yeah, like probably we do need a better, uh,
breakthrough. Yeah. Probably we do need another breakthrough.
I am saying that
these breakthroughs are not super easy to come by.
Dep- I think we d- just did the S curves debate. But yeah,
The one crux we have is that I am very uncertain of
you have somehow. You are somehow more certain in
have, and this is just a prior belief thing.
I don't know if there's, like, any evidence that you can show me.
[speaker_0] Yeah, I think there are-
[speaker_1] And maybe I don't know
[speaker_0] ... S curves we are tracking. One is like,
uh, curves for individual paradigms, and one is like some bigger curve
of, you know, like humanity's ML research as a whole.
So there, one is like the curve of pre-training
started saturating. There's a curve for
scale, when did they start saturating.
There's a curve for, you know, when did RL scaling start,
might saturate someday. And then there are multiple of these curves, but then
there's a bigger overall trajectory of how fast is humanity's ML
capability growing.
[speaker_1] Uh, that I don't agree with because
it. For example, if you start tracking it, there have been multiple AI winters and
AI summers. There were times where people thought
in AI, and if we just do work on GOFAI or we just do work on,
like, some other paradigm, we'll be able to get to ASI from here.
[speaker_0] Why aren't we including all of that as data points?
[speaker_1] So if you keep including them, then essentially, uh, this could
either be like a, a 1990s rush in
AI of, like saying that, "Hey, we have super intelligence.
We have chess playing AI, and we are, like, a few research ideas away from
having, like, super intelligence AI because we have Deep
we'll be like, "No, that, that S curve actually flatlined
super intelligence." And then you have a larger S curve of,
um, and, uh, transformers and R- RL.
And that S curve, depending on where we are on the S curve,
still saturate before we get to, uh, super intelligence.
Uh, and, uh, either that could happen, that is one world that could happen,
or another S curve gets stacked on top of it, and it keeps going till
we reach super intelligence. Uh, so, uh, or maybe, like, how close we
are to super intelligence, essentially whatever that bar is,
there or we need to stack more S curves to basically get us there
faster. Uh, I do agree that on infinite human
timescale, we'll get to super intelligence at some
Uh, but yeah, 2030 is the timelines that we're dealing with.
[speaker_0] Again, I didn't understand your argument.
If I track, you know, human- humanity's AI research progress since,
1970, yes, there have been multiple spans of few
years where we did get, you know, some one breakthrough, something happened,
then we had, like, you know, 20 years of nothing happening.
Yes, there have been multiple of these.
Uh, right now we could be in either one of
We might be about to reach an AI winter or we might be
about... Yeah, we might get a few more breakthroughs.
Those again might not, might or might not get to super intelligence.
Sure, uh, where are we in this? So far I'm agreeing it's now a question
how do you put the numbers on these things and...
[speaker_1] The, the quest- the disagreement comes from the fact that,
It's not a disagreement in the argument, it's a disagreement on how many
S, if a new S curve is needed or will these S curves scale to super
intelligence and how easy are these S curves to come by.
[speaker_0] So what would be a data point that would actually change
your mind? Like, for me, it's, like, fairly, yeah, almost obvious
that, okay, yeah, there is a b- huge backlog of, backlog of research
ideas that need to be, that will definitely try-
[speaker_1] That I think-
[speaker_0] And all of them require-
[speaker_1] Yeah, but-
[speaker_0] ... a lot of people to try and, yeah.
[speaker_1] Sure. I think sure there might be some research
overhang, but, uh, the probability of us
finding a breakthrough in the research ideas might be below, uh...
I think the ML research community is very, very smart.
Uh, they figure out all the best candidates
a daily basis, et cetera. And in the last five, six years-
[speaker_0] Yeah, I think that's where, yeah, like,
smart. It comes down to try, hit and trial random shit until it works.
[laughs] Like, I don't think people had, like, some, uh, brilliant insight,
"Okay, this is why this thing is definitely going to work,"
tried it a ton of time. I think people just, like-
[speaker_1] No, they did
[speaker_0] ... tried the random 20 random things to try
[speaker_1] If you look to-- No, but I don't think it
I think the people who try it have some intuition of, uh, why
this could work or why this wouldn't work, and they might be wrong or right and,
and the outcomes might look random, et cetera.
But the selection of which experiments to try definitely has
A good AI researcher is one that tries more successful experiments
And a bad researcher is they keep making bad bets on research
keep failing at it.
And, uh, my, the, my, my, the data point that
I want to look at is that despite so many,
so much money flowing into, like, finding research ideas, I'm sure
we'll be able to scale this further.
But is there an RL level or a pre-training level idea, uh,
already out there that has not been tried yet because of
compute? Because people, like researchers are literally drawing
whatever, um, a hat and implementing the idea as opposed to
reading the idea, understanding the viability of it working
Uh, I think, like, like I believe in the second one.
Uh, and if that is true, then, then where is the idea is my question.
Uh, and just because you got two ideas-
[speaker_0] Yeah, that question is definitely a crux like... Like yeah,
On one side you have like researchers understand nothing about the problem and
they're just brute forcing. On the other end of the spectrum you have,
researcher deeply understands the thing and they have an hypothesis
actually running the training run, they already know with confidence this is
definitely going to work. And you are saying, okay, researchers
end of understanding things. I'm saying they're far closer to the end
randomly brute forcing.
[speaker_1] I don't know. If you look, like, if you look at, like Noam,
heard Noam, what's his name? Yeah, Noam Shazeer talk about,
uh, when they were getting into transformers,
transformers out, when they were essentially scaling language models,
wrote the attention paper, et cetera.
Uh, it wasn't like a random idea that they had come.
Uh, Noam Shazeer has like this history of
ideas to try out. He's called like a, a magical researcher
because he can seemingly look at like 100 ideas and figure out,
like these are the one, two. He has like crazy intuition of these
ideas that could work because he understands these things much more deeply
average researcher.
[speaker_0] Okay. Oh.
[speaker_1] And there are the superstar researchers that can look at ideas
like breakthrough ideas much more quickly.And I don't think it's as
random as that, "Hey, let me just pick one and do it, and
Otherwise, I'll go pick another one tomorrow."
[speaker_0] For me, I see it more as, yeah, like, yes, Noam Shazeer probably did
have some intuitions, but also it was random that he
How would I put it? There have been three
Noam Shazeer directly contributed to one of them.
If you take any of the other research breakthroughs which Noam Shazeer did not
make, and you put him in one year before
happened and told him, "Look, here are all these hypothesis the different
researchers are making. Which one do you think will work?" I don't
have made that good a guess and told you, "Oh, this one will work."
[speaker_1] And I saw a Noam Shazeer, uh, and Jeff Dean
talk
about this exact thing, and from the story that they
their meeting perspective and understand, et cetera.
[speaker_0] Yeah, if what you are saying is correct-
[speaker_1] [laughs]
[speaker_0] ... there should be, like, the same researcher who's consistently
where the field is heading multiple times and should be multiple times-
[speaker_1] That's true
[speaker_0] ... not 100% co- correct, but, like, roughly able to see, okay,
are probably going to work and then actually roughly ends up correct.
Whereas I'm saying no actually-
[speaker_1] That's been true
[speaker_0] ... I'm saying no actually-
[speaker_1] Because the amount of, the amount of breakthroughs have come
No, the amount of breakthrough that have come from these superstar
like, very, very high. Why is Ilya Sutskever around all big
'Cause he has a crazy sense of which research ideas can work.
Why is Noam Shazeer around all these big breakthroughs?
'Cause he has a crazy idea of all these things to do.
There's a random reason why a random ML PhD you've never heard of comes up with a
crazy idea. It's mostly because you literally have to
uh, what's that guy's name, who's the GPT-2 main, Alec
Radford type level researcher. Apparently, Alec Radford has such a great
sense of what could work. He's like, he literally used to
do small experiments on Jupyter Notebooks, and
he then once he got convinced that this could work,
engineering to Greg Brockman or somebody who's like, "Yeah,
Just keep scaling it up. No, I'm sure it will work." Uh,
built so much intuition about, like, and what couldn't work
this crazy ML whisperer guy who can just, like, look at the
shape of the model and figure out, like, these are ideas worth pursuing, not worth
pursuing. And if you look at, like, Thinking Machines Lab, which is, like,
lab filled, filled with all these guys, John Schulman, uh, what's his name, Alec
Radford, all the OG co-founder guys, they have essentially had, like,
free rein to do whatever. And the best they came up with
that updated me towards that, oh, okay, there's, like, a lot of things to do here,
but there is no crazy research breakthrough paradigm that, that
oh, we got, like, a pre-training level paradigm
stack on pre-training and get, like, insane results.
[speaker_0] Yeah, I think I've identified a data point
on this. If you, uh, again, from these, you know, three or four superstar
researchers, if you're able to document a public track
of, well, uh, yeah, since maybe 2018
till 2026, some at least 10 years or no.
Well, yeah, at least, yeah, like more than five, at least seven,
consistent track record. Like, okay, here they made these predictions in 2018.
They made these prediction 2020. They made these-
[speaker_1] Dario is one guy
[speaker_0] ... in 2022, and they made these in 2024.
And, like, they didn't single-handedly do all the
roughly able to see where the next breakthroughs are going to come from.
If you can show me, okay, just the same guy
trend, then that would actually shift my
[speaker_1] I don't know. I think I 100% believe what you're saying
true. Uh, I don't know how... Like, I'm, I'm trying to think of what are ways to
show you this is happening. Uh, one way to do this would be that if you
look at the large breakthroughs, and you look at who's responsible
or who's close to those breakthroughs, it will seem like it's the same people.
And that should update you towards that, hey, how come, uh, Ilya
Sutskever is involved with all the big breakthroughs?
How come it came from the same guy who did AlexNet, is the same guy who did
GPT-2, is the same guy who did RL? Why, why is it the same guy who's doing all
these other things? Why is Noam Shazeer building all of these
Uh, why is Alec Radford building all these
It's because they have figured out or, or they have impeccable...
So they, they talk about impeccable research taste,
taste is what is really hard. And research taste is this intuition that
researchers have that can figure out from a pile of, like,
one to worth trying from the compute we have to get the breakthrough.
[speaker_0] Yeah, so literally what you said,
here is Alec Radford's track record of research hypothesis going back
entire eight years-
[speaker_1] But why won't you just buy Ilya's track record?
[speaker_0] Uh, I literally don't know enough about this
What did Ilya say in 2018? What did he say in '19?
[speaker_1] No, he doesn't say anything.
[speaker_0] I'm literally not saying-
[speaker_1] But basically the fact that he doesn't,
Basically, if he doesn't have to publicly make any of
there's a reason why all big breakthroughs are around one person, it
stands to reason that this person picks better research
person who's picking research ideas at a random, at random.
[speaker_0] No, no, but-
[speaker_1] Like-
[speaker_0] Was he personally the one who did the breakthrough,
to be at the lab where somebody else did the breakthrough?
[speaker_1] No, no. He personally was overseeing research.
He personally was green-lighting the experiments that he thinks would work.
[speaker_0] Okay. Uh, okay, then we can take literally Ilya Sutskever
example. Uh, which breakthroughs would you say, okay,
is significantly responsible for versus which ones you think he just happened
to be there?
[speaker_1] Deep learning.
[speaker_0] Sorry?
[speaker_1] Deep learning, he was-- Deep learning, he
[speaker_0] No, when you say deep learning,
Okay, I'll explain it. Cool. Uh, yeah, I agree with you. Fine.
Ilya was significantly responsible, a-along with other people, significantly
responsible for AlexNet, sure.
[speaker_1] GPT, the GPT ideas he was significantly responsible for.
Just training transformers. GPT-1 also.
The idea that we can essentially scale transformers or we can, we can find a
scalable transformer paradigm and build language models from it.
[speaker_0] Uh, okay. So which model was this? Was this GPT-1?
[speaker_1] This was the generative pre-transformer paper.
He is directly responsible there.
I think if we look at the GPT-1 paper-
[speaker_0] One second please. I know this, I know it's annoying to,
things middle of video, but, like, I actually want to now go read
is.
[speaker_1] Sure. It's this paper. I will send it to you.
[speaker_0] Oh, okay. It's in the chat. See, uh, when was this published?
[speaker_1] Five years ago. Twenty twenty...
No, earlier than that. June 2018.
[speaker_0] Is it June? You sure?
[speaker_1] This is the AI summary. Yeah. Yeah, June 2018.
[speaker_0] Okay, fine. Take care. I will buy that. Okay, fine.
Ilya has been there at two major breakthroughs. Fine.
[speaker_1] Then he's also been there at, uh, this thing, uh, for
the o1 breakthrough as well. Uh, Ilya didn't
green-light the experiment. Ilya was heading research that time and
green-lighted the o1 experiment. RL, basically RL scaling.
And if you look at Dario-- Sorry. Sorry.
[speaker_0] Dario, for this-
[speaker_1] Uh, he was at OpenAI at that time. He was head of research.
He was personally seeing all the AI research that was happening at OpenAI, and
OpenAI came out-
[speaker_0] No, but if he's head of research,
works in his lab, even if he does not want to, you know, do that,
hypothesis or, you know, prioritize it.
[speaker_1] You don't have to suggest the hypothesis.
I'm saying researchers are mostly the same
experiments, the best researchers.
The best researchers know which, which can control-
[speaker_0] But like OpenAI tried 100 things. Whichever of,
worked, he could take credit for it simply because
[speaker_1] Sure, but if, if there are 10,000 things that OpenAI didn't try,
if there are three big paradigm, three big breakthroughs
AI, and the same guy has been around for all
the set of the things he tried. It's the, the, the set is all the things that are
out there that he didn't try, and out of which he was able to freak out
or be around all three things.
[speaker_0] No, there are ways to be around the guy who
gives the correct hypothesis.
[speaker_1] But he's not like Sam Altman, who was probably around it, but he
committed to the research direction.
Anyway-
[speaker_0] Okay
[speaker_1] ... this is by the way, like even if you don't strongly believe
[speaker_0] No, no, this is very important. Like this part
Like, uh, was he the one who personally selected the
"Okay, this one is worth trying, we should try it"?
[speaker_1] Yes.
[speaker_0] Or-
[speaker_1] Yes, I, I did the... I'll tell,
But there is an interview of Dario Amodei
when he was working with Ilya Sutskever, who I think it's Dario
Amodei, who basically, uh, I don't know if it's that.
Anyway, it was, I think, one of these interviews
about Ilya Sutskever, and he's saying then he came,
research direction, saying that we need to do X,
we'll do this, and we'll do that. And Ilya Sutskever drew two circles,
two concentric circles. Inside he-- In one he-- And this was pre o1.
He wrote pre-training, and outside he wrote RL, and he said,
[speaker_0] Okay. Uh, if you can send me this, that will help.
[speaker_1] I'll find, to find that clip. I'll try to find that clip.
And this was like much before o1, when I think Dario or whoever
and, and the guy was like, "Okay, uh, it
makes sense." Like,
why am I complicating the research agenda that long?
[speaker_0] Okay, sure. If you send me this, that will again,
Like now you have given me three different data points, and Ilya was involved
directly, like not just like, okay, researcher overseeing, but he
picking the hypothesis and saying this will work. Yeah.
[speaker_1] Yeah. Yeah. I think I can to find... One second.
Let me just do a random cloud search to see if they can find the things.
Last time I remembered, maybe we'll find, but I, I've seen it and,
uh, provided this is true, uh, would you agree that research is
not as random as picking ideas from a hat?
[speaker_0] Uh, yeah. If you show me that, yeah, now three different
breakthroughs, uh, Ilya Sutskever personally was helping
pick the hypothesis rather than just happen to be in the same room or
overseeing the same lab. If you show this across three
that would tell me that there is something spec- some specific way
Ilya Sutskever personally looks at this problem, which almost nobody else in the
world has. Yeah.
[speaker_1] It's time to find out which interview. I think it
Anyway, cool. Uh, I think are there a couple of other things
that I think I disagreed with. Uh, so one was that research direction
there might not be as many low-hanging fruits as you think there are.
Uh, so 2030 might be in this thing. That was one crux he identified.
The other one was that, uh, intelligence is easier to build than
I thought. This is something that I think I've changed my mind on since
Ooty, uh, like since he last spoke, uh, about this.
It's basically the, if you've, if you saw the Richard Sutton
Dwarkesh interview,
uh, TCS-
[speaker_0] Yes.
[speaker_1] Do you remember what they spoke about?
[speaker_0] I think Richard Sutton's timelines were also something 25% by 2030,
remember correctly.
[speaker_1] No. He thinks it's more LL by LL-
[speaker_0] I am pretty confident Sutton had something like next five to 10 years
chance ASI. But yeah-
[speaker_1] But he thinks-
[speaker_0] I can't quite remember. Actually, you know why-
[speaker_1] But, but I also think that this whole idea that-
[speaker_0] Ilya's timelines, what are his...
Give me a minute.
[speaker_1] Yeah, and I also don't think this whole idea
agrees with you on timelines, it doesn't matter. Any-
Like it does matter what shape of beliefs he has and why he agrees to those
timelines and what shape of beliefs you have and why you agree to the timelines.
Uh, there might be a fundamental disagreement
You could update from some of his beliefs, not all of his beliefs, even though his
conclusion are the same.
[speaker_0] No, uh, like I initially started from a worldview
of
like pick even among the genius researchers, picking which research
hypothesis works is kind of random, and it requires just a lot of hit
and trial, and none of these people really know.
You are trying to update me more towards a worldview of, no, there are a few
genius researchers here who consistently seem to get all of the
predictions right. And I'm like, let's say I did update to your
worldview that all the more means I want to know, okay, what are these people's
timelines? If now you're saying, okay, I should defer to these people now.
[speaker_1] No, I'm sure. Go, go find out-
[speaker_0] I literally need to know what does Ilya Sutskever
Yeah, like-
[speaker_1] Go find their timelines. That's not the point I was making.
I was trying to make a point that you were saying that just because Ilya has
timelines and you have short timelines, it doesn't matter, uh,
what Ilya's arguments on research direction is or Ilya's time, like
Ilya's stance is on why and how we get this breakthrough.
[speaker_0] Those also matter. I agree.
[speaker_1] Okay. So it-- So cool. Find Sutton's timelines.
But the point Sutton made was that, uh, evolution actually gave us
language very, very lateUh, and most of evolution was
trying to optimize for things that we take for granted, which
uh, whatever, like being physical dexterity, et cetera.
And, uh, those things essentially, which according to you are part of
intelligence, those things actually took a, like,
time. Uh, and, uh, those would be very hard to do.
Uh, the second thing that he says that, uh, he thinks that if you can get
good at doing those parts, then everything else falls into place.
Uh, but, uh, the argument is if you can get to language, uh,
then getting to the physical stuff is easy.
And he's like, "No, most of it is getting to the physical stuff." And
then this language part and all of these part is something
[speaker_0] Sorry, I want to interrupt you because I feel
points we can both go check, and if we check,
our debate more productive. Data point number one of, you know, Ilya
Sutskever being personally present at all these three breakthroughs, I think you've
proved it for AlexNet, I agree. For the GPT-1 thing,
also kind of agree. More data would help, but I kind of agree.
The o1 thing, I'm not yet convinced Ilya
If you show me data, I can be convinced. Uh, that's one data point.
[speaker_1] Yeah. I think I found it.
[speaker_0] The other data point I want is, uh, yeah, literally what
What are Sutskever's timelines? Uh, and I'm saying, like, let's
first get these data points and then let's continue the discussion.
I think that-
[speaker_1] Found it. I think I found it.
[speaker_2] The people who are most responsible for that
and Jakub Pachocki. I think even like, uh, like,
uh, Dota was kind of-
[speaker_1] We can go like a few seconds before because he's talking about o1,
verify that he's talking about o1.
[speaker_0] And
okay, uh, I might agree with you by the way,
note, I'm like, why has this not been documented
there for all the three breakthroughs? That seems like a very big deal.
[speaker_1] What do you mean it's not documented?
[speaker_0] Why isn't there like either a Hacker News post or a Lesswrong post
saying here is the evidence Ilya was there at all three breakthroughs?
[speaker_1] But that seems, I think it's common knowledge.
I'm surprised you didn't, and I'm surprised you didn't
see this, but basically it's been talked about in a
Everyone knows that Alex Radford, Ilya Sutskever, Noam Shazeer are these
like insane superstar researchers who whatever they touch, whatever
ideas they pick turn out to be the right candidates always.
There's another one where Jeff Dean and Noam Shazeer-
[speaker_0] Okay. It's definitely not common knowledge,
to actually write this up and update a bunch of people.
Like, I think I'm not the only one for whom this
from you this. Yeah.
[speaker_1] Yeah. But, uh, okay, cool. Uh, I think I've-- The clip
something Claude found out. Uh, this is not the clip that I was talking about
originally, but I think this is even tighter evidence than what I
because he's directly saying that the reasoning breakthrough came from
Pachocki and Ilya Sutskever.
[speaker_0] Okay. Yeah. I mean, I will need a few minutes to properly go
okay, fine.
[speaker_1] No stress, no stress.
[speaker_0] Like for now I can maybe buy it. Okay. Let's, uh, buy this.
Okay, cool. Yeah. Also then I want to know, yeah, what
timelines then if you know that. Uh, any of the people you mentioned-
[speaker_1] We can just ask chat
[speaker_0] ... Alex Radford or Noam Shazeer or Ilya Sutskever. Yeah.
[speaker_1] Yeah. This is one post I found on EA forum, uh,
run by Lessig, uh, where they're talking about Sutskever's--
criticizing Sutskever for not having transparency on his timelines and
not saying why.
[speaker_0] I think you have successfully updated me a bit, well,
towards the idea that Ilya Sutskever personally, like not
even any of the other AI researchers in this space, but Ilya Sutskever
specifically has like great research taste and he's consistently able
to pick good research hypothesis.
[speaker_1] Good.
[speaker_0] Not yet shifted my timelines by a lot. I think that's where I'm at.
[laughs]
[speaker_1] Yeah. I have given you-
[speaker_0] Because yeah, again, like it's not obvious to me, okay,
if you told me, "Okay, update towards Ilya
than you," then sure. But are Ilya's timelines
[speaker_1] Ilya's timelines are--
His minimum is still longer than your maximum.
Anyway, maybe not. I don't know. And I think his definition of
might be very different.
[speaker_0] Fair. So yeah, now we will have to go more into
Uh, okay.
[speaker_1] We can end the recording there.
[speaker_0] Okay. Uh.
[outro music]