Uh, so in general, the trend has been, you know, once we see something happening,
we as human researchers think, okay, now how do we put a human expert into this
problem? But the way it actually ends up being
box brute force kind of approach where you, again, throw the human researcher out
of the loop.
So, that's the whole point of verification,
Where models can, like, run, but then you have, like, no way of saying,
running in the right direction or are they going in the place
And even where they want to go, like, we, we don't have, like, a good way of
setting the direction . Like, humans are giving those direction as prompts or
whatever, but-
But I think I'm saying the planning part is hard and the internal reward system
is easy, I feel, relatively.
Like you, you, you can create better players only
reality or, like, some grounding signal directly.
Like, it's just that, yeah, even now, like, if you can scale groundedness,
the problem will be cracked. And I think we just have disagreements on
scale of grounding
is.
Uh, hello, everyone. My name is Samuel Shadrack.
Uh, I have been following the AI space for the
past few years now. I graduated from IIT Delhi in
2023. Today, I have with me Bhishma Raj, who's a good
friend, who works at Google, who has also been following the AI space
some time. Uh, today we are going to have a debate on
timelines, uh, by what year we will have
what capability of AI. And we both have some
differences of opinion on this, and we will try our best to, you know,
exactly where we disagree, exactly why we disagree, and you know,
us resolve that
disagreement. Uh, Bhishma, would you like
to say anything more?
Hey, guys. Um, so I'm Bhishma. So I'm ...
Just want to add a small caveat before I introduce.
Like, all the views I, uh, express here are my own
and doesn't reflect my employer, which is Google.
So I've been as part of, like, uh, EA and like
rational display space for ... last from like 2020.
And I've been following how AI space progress.
I was initially a skeptic, but, uh, by the recent progress, especially
after 2022 and '23, things have been, like,
uh, advancing so fast. But still, uh, I do
think that we need to have like a more moderate position and, um,
we ... Like me and Sam had had like a lot of
before, so we thought it would be a good idea to
public and, um, just like work out our
differences and see where we agree and where we disagree.
So yeah.
Awesome. Awesome. Uh,
so I think initially we will just start by, you know, explaining our
position. You know, explaining any definitions,
reasons why we believe in our position, and after that we will just go
back and forth to identify, you know, the disagreements.
Yeah. For sure. Yeah.
So I think I'll be going first. Uh-
Yeah.
Okay. (laughs)
Should we start a timer?
Uh, yeah, we can do them.
All right. Think 10 minutes,
right?
Yeah.
Yeah, started.
Okay.
Uh, so
I think, uh, first of all, I think super
intelligence is possible to build.
Uh, when I say super intelligence, I mean an AI that is better than the
best humans at all tasks humans care about.
So this means, you know, any AI physicist that's better than,
Einstein or Hawking or any other physicists.
This means an AI mathematician that's better than Terence Tao
good human mathematician. It means
an AI that's better at customer service than the people who are really good at,
know, customer service and sales and hospitality and so on.
It means an AI that's a better software developer than, you know, the best human
software developers on earth. It means an AI that can do
manual labor also, an AI that can, you know, clean your kitchen, that can,
know, clean your dishes, that can, you know, operate factory floors,
operate vehicles. Just any task you can imagine humans
doing and humans caring about actually doing well, I'm saying, you know, we
can have an AI that will be better than, you know, the best humans at that task.
That's what when I mean when I say super intelligence.
Uh,
this could be an AI that is slightly better than experts,
AI that's way better than experts.
For me, both of these count as super intelligence.
Sorry, can you repeat?
Uh, an AI that is slightly better than the
and an AI that is way, way better than human experts.
for me, count as super intelligence.
Okay.
Uh, so I think first of all, this is possible to build in theory.
I think we might actually end up building this in practice in the next few
years. More specifically, by the year 2030,
which is five years from now, we are in 2025 right now,
I think there's a one in four chance, so 25%
chance, uh, humanity might end up building, uh, super
intelligence.
Uh, this will probably be one of the few AI
already,
uh, building frontier AI models. Like, I would be surprised if, you
know, some new AI company came up in 2028, which nobody has heard of, and
they managed to build it. Like, that is possible, but
to me. For me, a lot more likely outcome
companies. So this is, you know, OpenAI, Anthropic, uh, DeepSeek,
DeepMind, XAI, SSI. Just one of these companies that's already
built frontier modules, already has a lot of funding.
It will be one of them that ends up building it.
Uh, I have one caveat here, which is, I personally do not want
this to happen. And I'm also, you know, involved in efforts to prevent this from
happening. Uh-If there were, you know, a political movement
that could stop all these companies from building superintelligence, then this
probability can change. Like, I don't think this is, like,
This is ultimately something humans are choosing to do, I think.
Again, human effort can actually change this
I'm just saying, yeah, unless me and people like me do something to really
work hard and change this outcome, like, by default, I think this is what's
happening. Uh, now coming to
why do I believe this specifically, uh, I
have made, you know, some videos on this on my channel
but...
Okay, I think I'll... First of all, broadly,
inside view. Outside view is when you look at what other people who
topic are saying. Inside view is you understand the topic deeply
and-
Yeah.
... then you form an opinion. I personally give a lot of weight to my
like what is how I understand this, and not so much weight on, you know,
people are saying. But I also think it's useful to just look
other people who are experts actually saying about this.
Uh, from an outside view, uh,
yeah, I think many of the most famous
space avoid giving explicit probabilities, like they avoid giving-
(techno music plays)
... exact numbers, but they do broadly agree that AGI, ASI,
again, different people define it differently,
might happen in the next five years or 10 years.
This includes the heads of all the AI labs, uh, heads
of OpenAI, heads of Anthropic, heads of DeepMind,
uh, this includes the founders, at least some of the
founders of AI research including, you know, Geoff- Geoffrey Hinton,
Max Tegmark, uh, Yoshua Bengio,
uh, Paul Christiano who worked on the RLHF
technique. In general, yeah, a lot of AI researchers
There are also a few AI researchers who are on- of a different opinion
that, you know, this won't happen, uh, such as,
yeah, okay, such as Yann LeCun, such as Andrej
Karpathy, such as, uh, Jagan Sredibha,
such as Rich Sutton. There are researchers who have the opposite
too, that either this... Actually, no, sorry, I would
like to remove Rich Sutton from that list.
I think Sutton has recently said, you know,
so. But yeah, th- there are a few research, again, AI
researchers, uh, who believe, you know, this may not be happening
immediately, and they point it more as, you know, this is a far off problem,
but I think,
yeah, I think both positions are popular positions in the research
community lately. I think neither of these
either it's happening or it's not happening,
anymore. Uh,
coming to
the actual reasons from AI research which I believe
in, uh... So right now, there
are two clear paradigms which people are aware of, uh, first
is model scaling, the second is RL scaling.
Model scaling is you take a giant transformer, bigger than any
transformer built before, and you spend more compute, you use a
bigger dataset, and you train it. And initially
people started with text, now people are training it, you know, with images,
videos, with, you know, even just actions, like, even for robotics you can have
actual motor actions. So, at this point any sort of input,
you can use as long as you have a big enough dataset,
on GPUs. That's model scaling and that is how we have got most of the
progress of the past few years. RL scaling is a more recent
technique di- discovered last year, which is
working especially well on math and software in particular.
It's showing, uh, progress in a lot of domains but it's,
working well on math and software, which is you take a transformer and you do
reinforcement learning on top of it, and then you put a lot of compute on
this. These are the two known paradigms, there could always be an
unknown, unknown third paradigm someone might
that may be similar to these techniques or also a bit
different. I think we are probably going to use deep learning
going to use transformers to build ASI.
If it's built in the next five years, it's probably using transformers, probably
using deep learning, it's not some completely out of the left
field, you know, new way of doing it, but there may be a few tricks just like,
know, uh, how RLHF is, you know, one specific trick that was
used that made ChatGPT popular. There may be a few tricks here
are... ended up being in, you know, discovered in the next few years and that,
transformers, plus a lot of compute could get us there.
That is possible, maybe even likely for me.
Uh,
yeah, I think one last point I do want to bring
the difference between,
uh, yeah, uh, recursive self-improvement
and just, you know, serial and parallel speedup.
I think this is a point that (laughs) ends up coming up in our,
discussions when we have privately.
So recursive self-improvement is the idea that you will make an AI
better at AI research than human researchers,
you know, these AI training runs are worth doing, these ideas
Those ideas are not worth running, it will just have really good intuitions
will just do all the AI research. So you'll just have AIs building better
building better AIs
for, you know, again, many years and at the end you have, like, you know,
AI that comes out. So that is the idea of recursive self-improvement,
is possible, but that is not where most of my, you know,
probability mass of ASI comes
from. Uh, another idea
which is similar but maybe not same, which I have more probability
mass on, uh, which is
speedup due to serial and parallel, which is
basically, AIs think way faster than humans.
Like, if it takes me, you know, uh, 100 minutes to write
some interesting text, the AI can write that in one minute.
if you extrapolate that, you know, if it takes me 100 years to, you know,
new research, maybe the AI will do that in one year.
I think that is an argument which has more probability mass in my
which tells me that...... once an AI starts doing a thing
at all, it will start doing it, you know, extremely fast.
And I think just be- being able to do something quickly, being
able to iterate faster, get feedback from reality faster,
this could be in math, it could be physics, it could be,
lab-driven field like, you know, biology or, you know,
on. Uh, it could be in persuasion, it could be hacking, in any skill.
I think just being able to run more iterations, learn from what you did,
your failures and do that faster, I think that is also a point that,
seems important. I think these are the main reasons.
There are other, you know, smaller, minor reasons that may come up but I think
these are the main reasons I think superintelligence might come in the next
years.
Yeah. Anything
closing, remarks that you
want to make?
Uh, nothing much. Yeah, I think broadly there's, you know,
other people's
opinions telling me that, you know, this is not a fringe opinion, this
mainstream opinion now. Like, saying 25% superintelligence is no
longer a fringe opinion in the AI research community.
Plus, you know, all these technical reasons, you know,
uh, you know, superintelligence via speed, I think these
combined give me this, you know, 25% probability of it
happening by 2030.
Uh, makes sense. Yeah. So
should-
should we start or should I start?
Yeah. Yeah, you can go ahead.
Okay. Yeah, I made some notes, like, uh, on your,
whatever you said. You can also write down
in your section.
Yeah, yeah.
Sure.
All right. So, let's start.
So my... I think, like, we
agree more than we disagree, but I'll just say my arguments
for, like,
why, um, why- why I'm taking
the position that I'm taking which is, so my current position is
like, AGI by... Okay, I'll first define what like AGI, ASA
is for me. So, the way I see those, like I don't- I don't have a
concept of
just
single AGI because it's... Even like humans, we
are not like general, I would say like, in a sense,
um, like for instance, like we- we still suck
at like a bunch of cognitive tasks compared to like
chimpanzees or other animals. So
I- I find the metric that like just saying,
uh, the model is like, uh, good at
certain things or like better at certain things than human, I feel like
o- we already have that in like so many things
Which means that it's- it's entirely possible.
We can have a model which is like superhuman in like many things but
still, like, it may not like generalize.
Um, so
like my whole point is, um, I would say
AGI
is like, um, it's... I think
in terms of like percentiles where let's say, uh,
it's AGI 50th percentile would be like, okay, like the model
is, um,
it's- it's at least like in the 50th percentile
of like all the tasks that humans care
about. So,
uh, but then
it can still remain like bad at certain tasks.
Like I don't, uh, I'm not saying like it's-
it's like always superior in most tasks.
And, um, so in my mind, like AGI and ASA are
like, um,
are kind of related terms. Like I'm not saying ASA
will... um, that'll be like, it'll be like very spiky is my
understanding, like in terms of, um...
Like it- I don't see like a singular being which
um... which
is... it'll- it'll be like a
function of multiple things. Like I don't- I don't see
that, okay, it's just like one system
so many ways. The more I, um, look
at it and think about it. And I also want to, I think
decouple
a few things. Uh, I'm- I'm not
like overindexed on LLMs. Like
I've- I have a... I place like a lot of problems
that are, like, okay, certain things that we learned
in LLMs might help the fact that, um, yeah, you need like
parallel architectures, the fact that you can reuse like so much
data that humans have created. But
there are still like
a lot of issues that we need to figure out, like
we get to like more adaptive, um,
like systems which can,
it can actually like, uh, use like act in the world
and then use the feedback that they get from the world and just
like continuously learning about the world.
So- so I hope... like a lot
of disagreement I have with like your views just that,
um, like right now like most computers going,
investments are going for LLMs and, um, I mean, I- I'm
sure like they will progress and like, um, like they'll-
we'll see like a lot of improvement.
But then,
uh, the- the- the points that you mentioned, right, like that
they'll,
um, they can like, uh, they can just like
iterate, uh, they can- they can like
work on, I mean, they can take actions, they can improve on the
feedback that they get. I think-I'm not...
Yeah, I don't, I don't buy that LLMs will be like the right
call for that. So
yeah, like that is like the biggest argument
And um,
um, what was? Like RSI... So
I would...
Like my main worldview is just that,
uh, intelligence is like a very...
Intelligence in not just like individual level, but
somehow like we can have like these parallel in terms...
Like you brought the point that, um, serial
versus parallel thinking, right?
Like, and obviously like one, or like even in humans, like
I don't think like each individual human, even though like they're extremely smart,
like their capabilities and what they can
build is kind of hampered the same way.
Like, um,
um, you would, you would like want some notion
of, um, like a,
like collaboration or, um, some
like, (dog barking) like you, you just, like
need
a... What is
it? Like, I wouldn't say like need, but
in my worldview
it's o- (door slamming) it's about like how
like... Right now, like
with LLMs, like we, we just like, we don't
have, um...
Like, like, it's just like hard for
them
to be like, um,
to take a direction and then like sustain on the task, and
also
like
be consequential or like they, they just...
There's just like no
innate goal that they're trying to achieve.
(dog barking) So
in a way, (door slamming) I, I somehow
feel
like to do, to do the exploration or to do
the, to do the
self-learning capabilities, like,
you, you just like... your, like these models
need like an architecture which,
which gets those in, uh, in their drives,
which I don't see the current text-based models to be getting.
So I'm not denying it never happened, but there are like
these issues of, uh, how do you call
it? Like,
(dog barking) like them (door slamming) not
having like a
actual goal to be like pursued. I mean, yeah, you can say that, yeah, we have
agents and they're like, uh, taking these goals and stuff, but
I, I don't, I don't find it to be like convincing
it's just, I mean, like at the end of the day,
um,
like your, like your agent is
like... First of all, like they need to be like sample efficient
they
like, their goal pre- (door slamming) like the goals
being given are like too narrow in a way.
So,
uh, not, not just narrow, but
the models just don't have like, um, uh,
like a huge,
um...
Like they don't, they don't still have like generalized
to actually like interact in the world.
Like they are improving, but then
at the scales that we are talking about, like it's just hard.
So
yeah, like that's my
main issue.
(door slamming) Sample efficiency
and narrow goals. Yeah, I
mean,
uh, so the question is like,
um, is there like a connection between bad
narrow goals or narrow knowledge?
Um, yeah, I mean, so if your sample efficiency is
not great, then you,
you, you just like,
you just can't
make... (door slamming) You just...
It's, it's whatever. Like
I, I feel like the, like the space of
things you can just come up with is like huge, and humans are able to
make progress because we have like good priors.
Like, so each hypothesis is like
a... It, it just cuts through like so much
noise, and that's the whole way, like we are able to like build on
top and, uh, get ahead. And also, I mean the biggest
problem, at least I see with LLMs, is like the grounding problem,
like no architecture which, um, like...
So the LLMs, like they're... I wouldn't say like not character.
Like, you know, the grounding is like the training data, and
mediated through humans. Yeah, you can be like,
okay, the LLM will act on its own and it will create its own
data, but then, uh, how will you...
Ho- like what,
like what kind of verification will the LLM do to figure out that, okay,
hey, this is right and this is wrong?
So
yeah, I mean, I'm not saying it's not, it's impossible, it's just that
I don't... Like there are a lot of these issues where
like, just by scaling, just by scaling the models,
like- (car hooting) yeah, I think it's
10 minutes, but I'll, (car hooting) I just want to finish by
um, so in... even though I
agree with you in theory, like AGI is possible and
better than humans, like geniuses-And they can be
better than humans and, like, more stars.
And you say like serial, parallel, RSA
and RL scaling. Like I,
I agree in, in theory
but
it's just that I don't see it, like, with at least like LLMs
progress in a way. Like, even though it's surprising the w-
progress and I do think they will progress a lot, but it's just that
unless we have answers for this, it's, the progress is going to be like
very, uh, what is it? Like, it won't be well-rounded.
It will be, like, very one-sided. You know, one-sided is, as you said,
like easily verifiable domains and that alludes to the grounding, right?
Like, uh, if you somehow are able to crack
grounding, uh, yeah, I think it's been
shown that given grounding is somehow solved, the models will
just, uh, like, crash or, or
y- you can just, like, scale data and then things will just
improve. So yeah, but, but we don't know
like how hard
grounding is going to be. Like... And the other one is, like, high-quality
data. Like right now, the way the models generalize is
through, uh, more and more, like, high-quality datasets.
Like, either for RL or, uh, even for, like, final
instruction during SFTs. So,
like, are we bottlenecked by the rate at
compute? So yeah, that's,
that's my
main thesis. Because if data is the bottleneck then
it's... And it's not like just random humans.
Usually these are, like, high-quality datasets so they are like,
they are like bottlenecked by how many
high-quality people are there in this area
can answer and how quickly. So, yeah.
That's
it.
Yeah.
Oh. Got it. Got it. No, that was a great reply.
Yeah, I know I kind of rambled a bit in the
I hope, um...
Yeah, yeah. It's cool. I think I could reply to some of the
points you mentioned. They are new points that
things to say, which is good. Uh,
yeah. Okay. So I think the first major point you
(laughs) at least the models we have right now, they're not able to, like, just
keep one goal and stick at it for, you know, very long time horizon.
Like they can't just take a goal and be, you know, "I worked on this for six months
and here's what came out at the end." Like that is something today's models are not
capable of. And-
Mm.
Yeah. Factually, I agree that's true.
We don't have any model that can, you know, work for six months
or maybe even a couple of weeks. Uh, that is
true. However, I think the trend has been going in
direction.
Uh,
okay. No, I think even before that.
First of all, I want to talk about, okay,
humans want to build agents with long time horizons.
Uh, this is not a thing that is happening accidentally.
Humans-
Yeah, they want it. Yeah.
Yeah, yeah.
Yeah, I don't... I'm, I'm just saying the progress, uh,
like, um, like... how do I put it?
Like, the nature of its self-exploring or,
um,
it's, it's just that... (sighs) Like, I don't,
I don't see the current architectures like...
Yeah, I, I think it's just that initially I thought, yeah,
made sense, but now
I just don't... It's not like very obvious, like,
should happen. Like, I think the main argument is, okay, hey,
goal I give. I'll just be as stupid as fetch me,
And then the model will be just, be so consequential and it'll be like,
it can all create all these sub-goals and whatnot.
Um, I, I just feel like,
uh...
You know-
Sorry?
Yeah, yeah. No, I got your point. (laughs) I th-
different point.
Yeah.
No, it's...
No, I think you were talking about, okay, if we gave an AI a goal then
it will try to make, you know, sub-goals.
Uh, like then it will realize, okay, I need to have certain sub-goals.
Then it will pursue those sub-goals.
I think that's the point you are making and that is also true.
I am talking about this at a human level, which is humans
want to make goals with longer time horizons.
Like, so there's an AI thing happening where,
a- like if an AI could, you know, work at a goal for six months, it will
realize, okay, I'm having sub-goals like, you know, acquire po- acquiring power,
you know, self-preservation, et cetera are useful.
I'm just talking about the same, similar sort of dynamic
which is like humans want to make an AI that can, you
know, consistently work-
Okay.
... at a goal for six months. Uh, human-
Yeah, I think there, I, I do agree but...
So yeah. So for you-
I, I, I feel like... Uh, I'm really sorry to interrupt.
I feel like I should explain the full point and then-
Go ahead.
... you can give the rebuttal. Uh...
Yeah. Like, in the e- end right now it starts with, you know,
being like, you know, "Oh, it would be nice if I had an AI
you know, which physics experiments are worth running," or,
had an AI that could prove all these theorems and, you know,
paycheck and, you know, the AI will do all the work for me."
And the end goal of this is, you know, "I wish an AI could just run
this, you know, entire industry for me and, you know,
humans. That will be so much better.
I don't need to spend my brain power, you know, running this industry.
It will just run itself. I'll collect all the profit.
I don't need an AI to, you know, win elect...
Like, I don't need to sit and think, okay, how do I win an election?I'll just be
like, you know, there's AI, it will figure out how to win the election
it will go do that." Like, humans want to build these things because
themselves have incentives to try to do
things like this. And in fact, a lot of the money that
industry is explicitly with these types of
intentions.... in mind, like if any of
these AI companies are able to build a model that
can, you know, pursue goals long-term, that makes a lot of money for the AI
company even, you know, next year, next two years.
So I just want to say, you know, uh, from the outside, there is a lot of in-
pressure
on AI researchers at these labs to think of, you know, how do we make an
AI that can pursue long-time horizon task because if we do,
that means, you know, more power wealth for our AI lab.
Now, specifically, okay, all this researchers have pressure on
them to build this now. Just because they have incentive doesn't
actually be able to pull it off. Now, yeah, how will they actually
build this? Uh, then I want to now talk about past track
of transformers. So I wanted to talk about RL.
So there is a past track record of, you know, RL,
that. You know, even when, let's say, GPT-2 launched, so
GPT-2 was
the first AI model that was pretty good at speaking English.
Like, it would respond, GPT-2 or to some extent even GPT-1.
So GPT-1 was literally trained in, you know, a GPU cluster in, you know,
basement. Uh, back then, it was easy to train these
models and... Yeah, it could
understand English, but it could not really, you know,
understand the exact question you're ans- asking, you know.
Maybe understand why are you asking this, maybe, uh, think, okay, what are the
steps to solve this. It could not do any of that,
understand English-
Right.
... back then was considered a breakthrough
people had all these, uh, natural language processing techniques
know, n-grams, like bag of words, like these
dictionaries.
Right.
And-
Right.
Yeah, like parts of speech and all of that was very heuristic.
Uh-huh. Yes.
Yeah, yeah, like that was a very heuristic way of solving this
let's go ask the hu-
Mm-hmm.
... a bunch of human experts, "What does English look like?"
into a bunch of rules and then we'll do, you know, some brute force on that.
Like it was a mix of human expert plus brute force, but
when we had transformer, it became a little bit less of a human
problem, more of a just a brute force problem.
You did not need a linguist anymore, uh, to make an AI
model that could speak English at all.
Like forget about good, the fact it didn't speak in English at
Then after GPT-2 came out, people were claiming, "Okay, look, this AI model,
was the first few words makes sense." But after that, you know, the rest of the
para goes into like some random nonsense because, you know, errors keep adding up,
like every new word that comes out is coming out from, you know,
distribution. And somewhere, you know, some ano- anomalous token will come out,
there will be error. And once an error comes out, you know,
model has no way of realizing it has made an error, so it will just start putting
more tokens after that. Like if you just ask, you know, like, "W-
for me," once a few words there in those paragraphs are either they're just false,
they're hallucinations or they don't make sense
these errors will add up. Like people thought that was a problem,
and I'm sure people thought about various techniques on how we might solve
this. And
I don't know, I think Yann LeCun has a bunch of
and, you know, this is exactly how we'll solve this.
And how we actually solved this is, again, don't use any human expert, just, you
know, make the model bigger and, you know, hallucination rates just go down and the
model starts understanding more things like, okay, this is how a sentence,
you know, first sentence connects to the second
connects to your actual question.
Uh, so in general the trend has been, you know, once we see something happening,
as human researchers think, "Okay, now how do we put a human expert into this
problem?" But the way it actually ends up being cracked is, okay, some sort of
black box brute force kind of approach where you again throw the human researcher
out of this loop.
Yes, I think so.
And yeah, now even when
what is happening with RL... Yeah, and I think this will now
tie in with the grounding point which you mentioned, which you said is,
transformers don't have very high quality data.
The data they have is noisy, it is full of, you know,
hallucinations, full of lies, full of humans we don't know.
I didn't say that. That's not me.
Okay, okay, fine, I am saying it. Uh, so if you take
just like that internet scale data, like it is true that there
in there and a lot of things that are...
But the models still end up learning
accurate representations. Like, or, or we don't know
like... I mean there's a lot of... So these processo- like
these labs they are very opaque but there's like a lot of
filtering and data cleaning that goes, like
before, uh, these models are fed, right?
So and that's a problem, like any...
I mean, these can be said inductive biases
So, uh, I'm just saying like it's, it's much com-
low compared to let's say as you said like how earlier we were doing
NLP, right? Like we hand-tuned the algorithms
hand-tuned. But it has gone drastically low.
But still, it's, it's
still bottlenecked by humans like
doing the, like some sort of verification or like
the signal for what is important
is gatekept by human speed. And I
feel that will... like once you saturate on
capital, compute and electricity, uh, this is going
to be the bottleneck.
And
yeah, like, uh, but that's when the question,
like how, how much can you push with...
How much can you push with let's say 100X
compute, 100X capital, 100X
electricity? I, I just feel like, yeah, it can, it
can be pushed from, uh... Like probably
now it's like 5%, like in 20 or 30 it will be like
15%, right. For AGI. Yeah. Like-
Got it. Got it. Got it. So I think you're broadly agreeing with
prior-
Yeah, I agree. Yeah.
Yeah, yeah. Like my prior of... Okay.Just do a route first and some of
this problems will just disappear.
You agree with that, but you're saying you-
Yeah.
... won't be able to push it far enough to get
Yeah, it wouldn't saturate in, like, 100x or... I mean, in order of magnitude.
Like, we, we probably have, like, two or three.
I think right now the epoch thing is,
uh... or how much is it? Like can, can you just share that?
Like, I think there are, uh, I think
10, 10 power of 30 or 29,
uh, flops,
like, are you aware of it? Let me just quickly check on you.
Epoch, uh...
I mean, I saw it in terms of dollars, like, we have s-
like it was RL what at least public information we have scaled it to like
I've not seen any papers spending more than $100,000 on RL.
So
compute trends is growing
this growth
rate. Yeah, most, most compute is like
10 power 26, but this is graph three.
Yeah, on models we have spent, like the pre-training,
around 10 billion.
Yeah.
So, yeah,
and so 10 power 26 is, uh, is the high... highest we've gone.
So
I don't know, like, maybe we can scale it.
Like, yeah, I think the bottleneck will be like how
put. Maybe we can, we can ex- we can put some
upper bounds on some realistic capital, right?
Like, uh, yeah, now it looks like one trillion is,
uh, easily possible. So which is like...
so I think right now the biggest source worth 1 million, I think
or...
Like, what I know is for RL we have gone up to
For pre-training we have gone up to $10 billion.
Yeah, but-
As of today.
... also I think the other data point is usually I think the final
training run is like very d- dwarfed by the research
and-
Hmm. Yeah, yeah.
... R&D budgets. So, yeah, which is very hard to estimate,
right? So
I don't think... I mean, yeah, I, I don't wanna over index on that,
but
let's say like five orders of magnitude or something, like five or six, same
series run.
So
with that orders of magnitude
will be... yeah, we might get superhuman
and, um, what to say, like,
like math coding and stuff.
But,
yeah, unless you solve grounding and self verification
in some way, it just doesn't seem to
be the way. And
I, I, I do have a theory that
it might be possible, but it will just need like a lot of
time, compute, compute. So, uh, we might be in an
overhang. Like that's also possible.
Like it's just that, yeah, we just can't afford it.
Like most people can't afford it. But like if, like a billionaire
wants to just burn through so much compute and get this to run,
uh, they can, uh, is my, um,
guess. But then I don't know like how much,
like how much they would wanna
publicize these capabilities.
So that is also possible. I'm, I'm not denying it.
Like, so even if... so what I'm saying is I'm, I'm willing to hedge.
Just because we don't see it happening, uh, that might be a
possibility that it is still possi- it is...
the capability is possible, it's just that we don't know about it.
Like that is quite high as
well. Because I give like high importance to
time and inference time scaling, which probably you also will,
right? So-
Yeah.
... and how do we agree that, yeah, it is possible?
Because
I, I can imagine like
hundreds of thousands or even million dollar inference time
they just like keep running. And then who is willing to spend and for
what? So like there'll be like a long time where
is willing to burn through that.
So, yeah.
You got it. You got it. You got it. That's like two, three,
points here. I mean one is, yeah, okay, if we literally just take transformer plus
RL how we do it, you know, today and just put, you know,
that, does that get to ASI? I think, yeah, first of all I have some probability on
that might just be the actual path.
And
yeah, I know I'm sharing, you know, capabilities I deserve, but yeah,
there is some chance it might even actually just be
because-
Yeah.
... as
they said, you know, we have right now spent at least publicly
declared is they spent only $100,000, but privately I'm sure by now they have
already spent more because they have bigger data centers.
Like why would you not spend it if you,
uh, can? Yeah, you were asking, you know, where
is the money coming from? For me that is not a hard question.
For me I'm like, if you could get, you know, a solution to, you know,
novel math problems, if you could, you know, solve let's say-
Anything.
... all these millennium, you know, math problems, if you could, you know,
biology research, like
even just to have that name, like brand name that, you know, our
AI lab was the first, you know, do all these new biology discoveries,
math discoveries, that alone is worth, you know, spending at least a billion
dollars. Like considering-
Yeah, I don't... Yeah.
... market cap is, you know, like 100 billion plus.
So, like, I think they will spend at least a million
Yeah, right.
Uh-
I'm questioning the appetite. I, I, I'm just...
I just said the money thing to put a...
like that will be the constraint, like how much people are willing
to fund, right? Like hypothetically to...
like-
Yeah.
... cap on how big compute, uh, energy and
data center s-... in, in progress.
Yeah, yeah. So for me, I'm saying basically that
Like-
Right.
... I'm very, very confident the number is above $1 billion for sure, that they
will spend on RL.
Dip... Yeah. I agree.
And for me, $1 billion is already significantly ahead from what
publicly told. So that already tells me, okay, maybe there is
just some, some bottleneck there somewhere.
Maybe there's some trick they've not figured out that is preventing them from
spending a billion dollars. And maybe once they figure that out,
be able to spend a billion dollars.
Like, I'm actually confused, like, why have they not done this already?
Like, why are there not already papers saying,
billion dollars and either we got a result or we did not get a result..."
I think it's hard. Every one few million. There was also a recent RL scaling paper.
Okay.
Scaling loss paper. And they spent like few
millions, just to do the experiments.
Oh, okay. First of all, now that will be actually useful for me.
I will definitely go through the paper after-
You should have the links and...
Yeah, yeah. After the chat, for sure, I will sit and read that paper.
But even then, that comes to my point. Like, why is there only a few million?
Like 1% maybe.
Like, why have we not spent a billion dollars on
Like, the data centers already exist here, the code already exists, the data
already exists. Like, why has this not been done already?
Right.
That has got me a little confused, so okay,
or something is happening, or maybe they just got negative results
willing to tell, you know, we tried this and it didn't work.
Like, that is always a
possibility.
So yeah, that is my first route to how we get to
not my only route. The broader intuition I was pointing
at was not just for, okay, you know, we will do RL and we will get ASI.
The broader intuition was, uh, if you have to bet
on two approaches, one is we, you know, hire a bunch of human experts and we throw
the problem at them. Option number two, we hire a number of, some AI
researchers and ask them to, you know, find some brute force way of cracking the
problem. I would generally bet a lot on the second
So even when you talked about grounding, you said, okay, what we need actually is
high quality data sets. We need human experts, you know,
Like, don't just take the entire internet
Get, you know, humans to, you know, filter out some subsets that are useful.
Maybe even pay, like some actual, you know, experts, like mathematicians,
physicists, whoever, and ask them to actually write down,
know, code or whatever. Uh, and right now, that is still
part of the loop. As of today, yes, we are still reliant to some extent on high
quality data. I'm just saying,
again, my intuition is the way you would-
(laughs)
... try to crack that problem is not by hiring a lot of human
experts and paying them a lot. You will first try, okay,
force and, you know, again, make the AI model crack the thing instead
to crack this grounding problem? Uh, another
intuition I will point out is that humans also do not get very
well-grounded data. Like if you're talking literally from you start
infant and you know, when... The knowledge an infant has about various
versus, you know, you ha- knowledge a PhD expert has in various
humans are also being consistently thrown very low quality data.
Humans need to form their own mental models to reason through, you know,
is useful, which data is not useful.
And humans spend their entire life, you know,
for, you know, the world is throwing so much information at me,
junk. How do I actually filter what's important
That's a important part and you need actual world models to do
human. Uh, most humans understand
English or understand some language, but very few humans are, you know,
physics experts or very few human, uh, you know, expert mathematicians.
And there are multiple reasons for that.
You know, humans are not so motivated to become an
That is also there. But there is also a grounding thing here, which
is, yeah, f- somehow, for whatever reasons,
evolutionary, whatever, humans find learning language to be easy.
But actually becoming an expert in some technical field requires
them to, you know, grind in that field, go through a lot of, you know, bad
papers, do a lot of bad experiments, try a lot of things that don't work.
Yeah.
And from all of that knowledge, they eventually start developing better
better priors, like, okay, you know, these are the types of experiments
worth running. These are the things that are not worth doing.
You know, these are maybe the bad incentives
written, and this is why you should ignore them.
Like, people build these mental models, they...
and-
Yeah.
Yeah, I think my point is, yeah, we might soon end up with AIs who are also
starting to build these higher level mental models.
Like earlier, we started with this very low level mental
does a paragraph that is coherent look like?
Then it's like, what is a mental model?
Like, what are mental models for just actually
Like, that's actually still not... What's a mental model for writing code?
And now that's being done, and now we will do even more higher level
what's our good mental models for just looking at all of reality?
What are good mental models of learning any scientific field quickly?
I feel like more and more high level, more general things we might crack through
very brute force approaches. Uh, you also mentioned that-
(coughs)
... how much capital we will throw at this
maybe this can be done, but we just don't have enough money to do
it.
Yeah.
And yeah, even in that, I just want to bring up the, what if we
have a new brute force approach?
Yeah.
Like, even transformers for that matter, like I would not be surprised
if you can take literally a fully connected network, like not even transfer,
fully connected network. You throw enough compute at it,
It's just-
Yeah.
... that amount of compute is very infeasible.
We had to invent something better called transformer to do it.
And-
Yeah, and even transformer, I think like MLP is like a huge
chunk of... I mean, a lot of compute goes to MLP
yeah, the attention layers are...
Uh, I mean they are
important, but that's not the crucial part.
Like the MLPs are also very, uh, very much ?
Got it. Got it. No, so you do broadly agree with me.
Even once you build transformer earlier, there were a lot of pieces, some pieces
Yeah, I, I agree, but it's just that-
Yeah.
I mean, my intuition is,
um, yeah, like we thought these were hard problems, but then the existence
of-... as long as they got these kind of stupid,
stupid, but very embarrassingly, like there's this strong embarrassment you
lately. So in a way that it's, like, like we didn't have
to
invent like such crazy stuff. We just had to throw a lot of
data and compute and keep the algo mostly
stable, and the models are just improving.
But then, uh, I think,
like
I'm just not, uh, how do I put
it? Like the rate at which they get more general
is kind of now bottlenecked by like
how, like you're, you're saying like, uh, even with a fixed
dataset, uh, just throwing more compute and,
um, just more, more, and more compute will just get you there,
what you're saying. But-
I'm saying-
... I don't know-
Sorry, so to, more compute and maybe RL gets you there,
some slightly better architecture, but yeah,
compute means the AI can now figure out for itself all these
things.
Yeah, that's where it kind of breaks.
So even you hinge on, like not humans, like at some
point the handover should happen, right?
Like, uh, where, okay, right now we are improving each iteration, like GPT
n, n+1, but at some point, uh, okay,
even now they use AI, like maybe for data filtering they're
or, uh, like it's, it's kind of used somewhere in the
the model development pipeline. Like they're u- they're using,
code, like some analysis, like certain snippets, they're using the
LLM to just, uh, use it. But I, I feel
like how is this handover gonna happen?
Like without these models, like, like will
it... Like as you said, like they need to go like higher
abstraction, which means that, yeah, like
they, they have to like propose like
better
ways of improving themselves and then going about and doing those
improvements. And we do see those things happening for a
bit. For instance, like, uh, the recent GDN
paper, like they released that, uh, it, I think it came
up with some
new drug or whatever, like, um, yeah, so
it is
kind of possible I guess, like I, I, I am seeing that I've used some systems
that they're able to propose, uh, solutions, but then
yeah, again, it's like a verification bottleneck rather
testing compute thing where okay you propose this, but
then how are you gonna verify all that?
Like, uh, what are you gonna
do to verify
all that? So
yeah, I mean
I think even here we agree. It's just that
I, I, I feel like the verification problem might
be like harder than we think it is. Like it's, it's...
Like
I think it kind of the crux is that
the generalization bottleneck will kind of always be there.
Same like how we had like superhuman chess for like, for like decades
and only now we have a general model.
Um, the same way we'll have like superhuman
thing. But um,
so I mean, so the way generalization is related to verification is like
you,
for the model to actually like generalize and like be self-learning
it needs, like it needs to kind of discern, okay,
is false. And uh, without any discernment, it's, it's just not
gonna like form representation. Like what is a representation?
Like, like,
like a representation is like, um,
like it's, it's like, it's like a,
it's like a thing which divides multiple
things. Like saying, "Hey, if you give me this object, it
is, I'll put it in this category." That, like the simplest ML model
is like a classifier, right? Like if you, if like why do you
say like a model can get like a 100% accuracy in training
data? Uh, but it just like memorizing it.
So like
it's, it's not, like only if it's,
if it can like actually like,
like verify whether its predictions,
um, match reality, it can like
generalize from those
concepts. Like or else it's, it, it has like basically
like it won't know whether it's in a training or like it's an actually
deployed
state.
Like I think it fundamentally comes
like, like
I think, I think yeah it's more like a
as humans we have the survival drive, like our, um, like a
teleological thing where
we have this purpose of okay, so we will,
um... So in a way,
I, I think like without those drives, the models, even
though they might be capable, they'll just be
like, they- they're just gonna,
they might not, um... I don't know
how you're going to bootstrap this verification and get them to generalize
better. So yeah, maybe people are trying
to, uh, maybe yeah, maybe people will just feed it that,
hey, uh, don't allow it to be sh- allowed yourself to
be shut down or they'll give these tries and who knows we'll, uh,
get those, I don't know. But
uh, un- unless we do that it's,
it's, it's kind of,
um... Yeah, like I just feel they might
bottleneck in, like a few key areas is what I
feel like.Like, it's not like ... Again, I don't
see, uh, I, I won't say, I'm not saying it's, it's not
possible. It's just that, uh, how do you, how do you
actually ... So right now the, I mean, it's pretty much the serial panel, right?
Like, the models can
do
extremely fast parallel, um, like they
can spawn, like, so many of these copies in parallel,
serial, it's fast. But now, the bottleneck is
not the speed. (car horn honks) Like, uh, even if I had, like, that
much, unless I know, okay, hey, this is what it is
true. Uh, only then I can just build on top.
Or else I'm like, okay, I'll just go in this direction for, like,
And then, like, some human has to, like, verify in some way, and then make
some updates. So, uh, uh, that's the whole point
of verification, right? Where the models can, like, run,
no way of saying, "Are they running in the right direction?"
the place that they want to go?" And even where
they want to go, like, we, we don't have, like, a good way of setting the
direction. Like, right now humans are giving those direction as prompts or
whatever, but, like, ultimately at some point the
direction will be, like, self-created or like, um,
so
yeah. I, I, I feel, I feel like that will be the bottleneck.
Like, grounding and, um, like,
verification.
Uh, okay. Got it, got it. Uh
...
Like, or else it's just gonna, (smacks lips) like,
yeah, like, I wanna ask a question to you, right?
Like, how do yous think,
um, like, this ... So, uh, the human
grounding and verification, how is it gonna scale?
Like, uh, or, or, like, I'm not saying it's
impossible, I'm just saying that will be the bottleneck,
timelines are probably served with lower. Yeah, I'm just saying that, like...
Yeah.
Like, maybe, uh, we, it's, it's, it's not like that
hard. We just need to churn more. Maybe we are, like, 70% there
and only in 30%, and if you crank for, like, next five years we'll get, like,
enough generalization.
Yeah.
But,
but I mean, like, look at the orders of magnitude, right?
Like, we had, like,
crazy orders of magnitude of texts from
internet, pretty much, like, starting from humanity.
So whatever we produce in the next few years is gonna be, like, a blip.
So
yeah, not, not sure about that argument.
Like, what
kind of text ... Yeah.
Got it, got it, got it. I mean, yeah, again, I do have couple of
points there. Uh, where do I place
this? Yeah, I think first I want to again emphasize
humans, which is that humans also don't often have
verification and-
Mm-hmm.
... humans end up learning tasks-
What we do-
... despite
... Sorry.
No, we do have, like, we
... Like,
uh, what do you mean we don't have? Like, we, like us,
like, we have feedback. I mean, because we are embedded creatures, right?
Like, we, like, anything we do have certain feedback
and, like, uh, it's not just embedded in the sense like,
senses and also our emotional, uh, like, amygdala
and stuff. Like, like, somehow, like, evolution has baked in those value
functions which
are able to give proxy, uh, proxy
signals.
So-
Yeah. No, I, I think that-
That allows us to get, like ... Like, I wanna, I want, like, that gives you the
drives or direction and then I, I feel like the
neocortex is like a secondary to the primal, like,
uh, uh, systems, like amygdala and stuff.
So first you have, like, the emotional,
uh, uh, like, those innate drives.
And once that, that allocates the value, like, that, that kind of puts,
like, a boundary saying, "Hey, out of all the things you can do, like,
like, high val- value." And then
now there's, like, a hierarchical planning,
the learning agent does.
Mm-hmm.
Like, but you're saying, like, somehow the neocortex will just exist on
like, which I'm, I'm ... I mean,
right now the way it is, like, uh, the neo-
It's like a virtual reality.
Yeah, exactly. Like, right now the AIs are, se-
an extension of neocortex. And neocortex as such doesn't have
the innate drives. It's, it's more like just a,
uh, I just give it any goal and it'll just be like, "Hey, okay,
fastest way to go from here." It's, it's more like a
GPS algorithm or something. Yeah, but it's, it's generalizes.
We can abstract away this kind of, uh, solution finding
across multiple domains. But in a way
still the neocortex doesn't come up with its own values.
You can just give it, like, some scripting thing
just find the way. And right now the models that we have
like, an extension of neocortex rather than,
uh, this emotional or, like, value,
um, uh, value. I mean, yeah, they, they don't have their
... I mean, the goal, even the goal direction you
say, it's somehow like, yeah, we are, we are c- again, like,
Like, it's, it's, um
... Yeah, that's what I meant by the handover,
Like, when will that handover happen and, uh, you just have this notion of
self-reinforcing, uh,
models.
Uh, okay. Okay, no, because you've taken
this whole thing in a very different (laughs) direction than where I initially
wanted to take, now I'll respond to your point.
Uh, yeah, I think you're talking about specifically
brain man- the human brain manages to crack this,
okay, it's not just that, you know, human infant learns all of these
things. Some of this thing is baked in evolutionarily.
Like, even by the time you have a human infant,
of this problem has been solved.
Yeah, I, I think, like, most are solved.
Like, we, we are more like a, just an implementation detail
of-
Got it. Got it. Yeah, I think the point of disagreement with
neocortex thing is, I think, the amygdala is kind of like the ...
Well, first of all, I don't know a lot about exactly how
think what you're basically saying is
the motivation is coming from that, but th- that isn't necessarily a lot of
high-bandwidth information. Like, if I say, you know, (phone beeps)
I am a, whatever, phys- undergrad physicist, I will now spend,
years of my life, you know, learning how to get good at physics.
Like, you're right, somewhere there is an evolutionary drive that
that I need to consistently keep working at this path
for, you know, many years, you know.
That's true, I can see that. Yeah.
Maybe, you know, it will (overlapping dialogue)
More safe, you know. Mm-hmm.
Uh ...
Yeah.
Yeah, yeah. You know, people will respect me. People love me.
Or, or I might be curious.
Yes.
There may be all sorts of these sort of drives that are coming in, and those are
actually driving me to, oh, now I need to, you know, spend my nights, you know,
going through physics papers. Like, that drive is coming,
huge amount of information, uh, specifically when I'm talking about how do
good at physics, like, most of the cognitive processing required
physics is not happening in that part of the brain.
Yeah, that's true.
It's happening in neocortex, I think.
Yeah, yeah, but I'm saying, um, like,
I, I ... Okay, yeah, yeah, I agree, like, a lot of it's coming from there, like,
it's ...
Yeah, and ... Sor- sorry, can I just finish?
Yeah, finish your answer, yeah.
Yeah, and even here, I'm now talking about the reward signal,
that is actually i- important. Okay, so there's two things.
One is, okay, h- did I actually consistently pursue physics
years? And second is, did I get good verification,
actually learning physics versus when am I not learning physics?
I think the, again,
more emotional limbic part of your brain is providing signals saying,
"Okay, this is important, care about this, work hard on this." It will also tell me
when I have worked hard, when I have not worked hard.
That, "Okay, now I spent the last few days working on
even if I didn't learn something, at least I tried hard." But
brain is not giving me signals on when am I learning physics versus
learning physics. That signal is still coming from the
neocortex. Like,
uh ... but that signal's still coming from the
that, okay, now maybe in ... I ran a few
experiments and, you know, the last three months, I
but now I was able to achieve them, and here is the model I used to figure
out what experiments to run. All that thinking is happening in the
that is what is telling me, "Okay, now I have actually learned something
in physics. I did not, you know, waste the last 2 or 3 months of my life."
now I think I also want to go a little bit tangent back to the earlier grounding
thing you talked about, uh, how do you get good
grounding, good verification. Uh, one way of solving
that
is,
yeah, you have human experts give the grounding.
You have, like, an actual physicist come and tell you,
right, you're doing this wrong." Another way
put the AI actually running a physics lab, just let it run its own experiments.
Maybe initially they're really bad, but then some of those experiments give a
result, and then it gets feedback, "Okay, these are good. Do more of this." You ...
maybe you collect some datasets like this, there're a lot of experiments that can
get feedback that way. But yeah, here I again want to point out,
you know, how humans learn physics is not this way.
Like, human physicists don't necessarily have
Human physicists don't necessarily ...
Like, they have a lab to use, but again, often the things they're learning, some of
it they're learning from running their own experiments, some of it they're just
learning by reading what other people did.
Like, humans also go through a lot of, you know, brute force and go down on, you
know, bad trajectories, and finally somehow they get it good.
Like, humans are themselves giving the reward
themselves (overlapping dialogue)
That's true.
... by doing a ... a lot of this sort of predictive kind of
where
I make a mental model of, "This is why this experiment should work,"
for it.
Yeah.
It does not work, and then I'd be like, "Oh, maybe this mental model is wrong.
Maybe I should think of it some other way."
Exactly.
Then I think of it some other way, then I do some experiment, then
works. Then I realize, "Oh, okay. Maybe these mental models actually make
sense." Like, humans are doing a bunch of this, so ...
And I think, again, all of this is happening in the neocortex.
This is all,
like, you know, which men- physics mental models are good versus not.
So, yeah, I feel this is a separate thing.
And I think in AI also similarly, the AI making guesses
of, you know, "I have some guesses on these men-
sense, but als- also not sure, and you know, let's test it out,"
you know, bad signal. I get, you know, a reward, and then that tells me, you
know, internally I should reward myself, you know, have the other mental model,
then I get some external reward. Like, there's, like ...
Models will start having, you know, internal reward systems that are
not matching 100% with the external reward system.
I think that also makes sense, and I'm expecting that will happen.
I just feel
... Yeah, yeah, I get it, but then it just feels like the neocortex
inner loop versus the outer loop. Like, in the sense that,
uh ... Like, you can think of, like, a brain as, like, for all the
possible things I can do, uh, I'll choose first one and then
I'll see how I can get that thing done.
Am I right? But then, which means that the first level is, like, of
all the things I can do, I need to choose.
I, I need a way of ranking or, like, saying something, right?
Like, I can, I can just now just leave everything
So, it, it just feels like the outer loop or,
assigns value is done by ... not by the neocortex.
Neocortex is m- more like, hey, once someone has given
me this ordering, like, how am I ...
from my current state, how am I going to that state?
Like, so the delta maybe it does, and then finding a path
between your current state to that new state.
But the new state is, is not ... I don't think it's a neocortex
thing. Yeah, you, you, uh
...
Yeah. No, I disagree with that. Uh ...
Okay.
How do I put it?
It's okay.
Yeah.
So I think there is the, first is the capability of doing the task,
there's, like, an internal reward system at least in
And I am not an expert in neuroscience.
Yep.
So you can feel free to correct me on any of this.
But, uh-
No.
So first of all there is the others' cognitive processing to do
like, okay, you know, this is... Okay, today I need to write a paper.
Today, I will write a paper. Or today I need to, you know,
will go meet XYZ people. And then there is another higher level, you know,
planning kind of capability of, okay, should I spend today, you
know, meeting these people or should I spend it, you know,
paper? Or should I just, you know, spend today, you know,
getting close to burnt out? Maybe I need to, you know,
Like, there is some high level planning kind of
is also happening in the very same neocortex doing the lower level kind of
Like, those two are being done by the same part
Yep.
Now separately, there is an internal reward system which
emotions, which makes me feel good when I've done things
I've not done things. And so-
Yeah.
... that internal reward system is a separate thing.
I'm saying the planning part is hard and the internal reward
system is easy, I feel, relatively.
On relative terms, like, how do you solve planning?
Like, how do you take good decisions on, you know, what is worth doing, what is...
Like, if you have a goal, you ha- already know the reward function,
need to make an actual decision, okay. Here's the reward function.
Out of these, you know, action trajectories,
Like, that is a hard question. But just building an internal reward
system from an external reward system, I think that is a relatively easier thing
solve.
No.
Because, uh, like, there isn't a lot of information
contained in that, that is actually really dumb,
I, I wouldn't... Actually, I mean, the evidence points to the contrary.
Like, if you can somehow design a good reward,
function, the problem is, like, done. Like, it will be, like, someone's...
Like,
uh, that's true for AlphaGo or, like, uh, which are, like, closed systems.
But even for RL and, like, math and coding, like,
once the reward is fixed, yeah, the models just, they just, like,
crush. They can easily... I mean, you just throw.
Which means, like, right now, uh, it's...
Yeah, exactly, that is the whole point of the, um,
verification or, like, how, how do you give this innate goal direction
really. Um, yeah, that's, that's...
I, that's what I feel the bottom line is here.
Yeah, I mean, no, doesn't that come under my point?
(laughs)
I feel like you're agreeing with me.
Like, I am saying-
(laughs)
... "Okay, look, we have tried RL for only one year
year. We have not actually even spent a lot of
saying here are my priors for why this is not hard, a hard problem to crack and
we'll crack it." And you are agreeing with me.
Like, if we figure out a good reward function,
Yeah, yeah, exactly. Like, uh, but then you are not saying anything about
hard... Like, your, I think your point is finding the reward
thing would be, like, easier. I'm just saying it might be
harder,
like...
Yeah, because I think it will be really simple.
Like, we don't need to do all the internal
of complicated.
Yeah.
Like, we need, we need to do an external reward thing
okay, when, you know, either this human verifier comes or,
automated verifier comes, you know, we need to do a gradient descent update
Like, that external reward thing we need to
some dataset. Uh, the internal reward thing,
code all these internal rewards of, "Okay, here I guess I did well, and here
I should feel good, and here I should not feel good." For a
again, I think we're just going to have some deep
Maybe literally transformers, maybe some other deep learning,
scaffolding on top, and we'll just brute force, you know,
reward thing. It will learn for itself, you know, what internal rewards to give
itself.
Um, yeah, I mean...
(sighs) I mean, I, I do agree,
like, in the sense that, yeah, it kind of boils
down to we were, like, very early in, like, the RL,
like, figuring out the good reward engineering
But then, um, yeah, like,
it's... Even the reward engineering
is, is not... Like,
yeah, I mean, uh, uh, the closest analogy I can think is, like,
like, uh, living versus dead. Like, that's like the reward I guess.
Like-
Yes.
And just starting with, like, the very simple, uh,
selection mechanism. Like, it's like, yeah, re- rejection sampling.
(laughs) And, yeah, which is that and, like, so many
parallel and, like, uh, serializes horizontal... Uh, sorry.
Um, serializes, like, parallel computation.
Like, we were able to, um,
like, yeah, like, build complex stuff.
But
I don't know. Yeah, maybe
that can be like a... So in this way,
like, um, like, the individual drives and everything just
emerge out of this training algorithm.
Like, it didn't, like, it d- like, the first you carry out or
something like, they didn't have to, like, be
optimizers. Like, they, like, the optimizers just emerged out of
it. But, yeah, I mean, I, I just
agree where do you test. Yeah, maybe all this, like, it's, like,
from a scale of... Like, (laughs) we are, like, talking in terms of,
like, like, error bars which don't matter from, like,
uh, uh, what is, like, humanity or even, like,
Earth's perspective of, like, evolutionary timescales.
So
yeah, I mean,
I- I'm just saying maybe you're, you're, you're overlooking on the
difficulty of how things are just because of the recent
progress
is what I feel like.
Yeah, I think I agree with you that a lot of recent progress has pushed
prior very high for-
Yeah.
The moment I see sparks of something, we're just going to crack that problem
completely. Like, the moment I start seeing this problem
starts getting solved, like, I actually see people made a dent on this,
"Okay, look, in two years this entire thing
will be, like, be nonexistent in two years."
Usually the way that...
... my goal is very high prior, and that's why I'm, like,
I've been start seeing, okay, like, these AI models are starting
to follow, you know, the same goal long term for even 10
And the fact they could do that, that itself tells me, okay,
internal reward where consistently for 10 minutes they could go at it, go down, you
know, multiple bad trajectories, and actually look back,
thing. Like, the fact that there is initial spark
plus, you know, in last historical, you know, at least five to 10
years of email research, whenever there was a spark, you know,
you know, two years later the problem got cracked.
That's why my prior for even right now the spark is there, then we'll crack it is
very high, even though, like, I don't know concretely whether this will
done.
I
think
our crux, ultimate crux is, like...
Like, I think we both even agree that the bottom line will be, like,
Right? So there are two. One is, like, yeah, we figure out, like, a
general reward mechanism, like a self-rewarding thing,
on, uh, like organisms looking after their own
survival or something like... So we stumble on some algorithm like
will give those innate drives. And the hope is, yeah,
SGD or something like... Yeah, the...
So, yeah, someone like you, you think the MISA optimizers will
like, extremely capable. Like, just SGD on some
crazy objective or whatever, like, with some crazy dataset will,
can lead to... That is one part.
Yeah.
The second part might be
just this grind, like, where keep the same thing.
It's, it's mo-... It's just a matter of time
exam-... Like humans, um, giving more and more
varied, like, high-quality datasets.
So these two parts, and, like, maybe third are known or unknown part.
Like, so, uh, I mean, this is what kind of we kind
of, um, narrowed it down. And maybe, yeah, we can put, like, some
numbers on what world we would think we have, and I think
that might help us understand why we come to these
numbers. Do you, do you think, like, maybe I should just write it down,
like, uh...
Uh, yeah, that makes sense. Uh, do you want to close?
Do you want to continue?
No, you just, like, write it down, and we can finish it, like, uh, so...
Okay. Yeah, yeah, I think I broadly agree with you so far.
Like, we are agreeing on these are the questions that are important.
And, yeah, we have not yet resolved, like, why our probabilities on these two
questions coming out to be different. Yeah.
Hold on. More like algo H 10. Um,
okay.
(keyboard clacking)
This is one, and then,
uh, the other one is
just grind and
collect.
Is it okay if I bring up one more data point?
Yeah, yeah.
Yeah. No, just
that at least in very narrow domains, like, you know, various games, like, you
know, poker and StarCraft and all, like, we have built models that can
pursue goal over long time horizons, so.
Yeah. I, I, I totally agree with you.
Like, see, the whole point I'm saying is, like, also there's, like, a outer loop
and inner loop. There's a outer loop is like, okay, I come up with a world, and
I say, "This is how the world works," and now you put, like, a,
some optimizer in this world where things are grounded.
Like, in a game, I know what a win state is.
I know what a, uh, like, what is called, what it means
to... So this is exactly what I said, like the amygdala, like, value system
is created by human. So once the value thing is
created, just reaching out those goals, things will be crushed.
So the whole point is, like-
So, um...
... how do I now just generalize these?
How, how do I make a
algorithm which will just spit out these value functions for arbitrary
domains?
Yeah, no. So in the chess example, uh, what we're...
Like, we are not saying, "Okay, these are the rewards of, you know, chess win
state, chess lose state, and now, you know,
happening is we put the reward on chess win state, chess lose state.
Yeah.
Then the model has, on its own, learned a bunch of internal rewards
this state is closer to a lose state than a win state, and therefore I will
internally give myself this reward on this state." And it has learned, you know, a
bunch of these rewards internally.
Then given all these internal rewards, then it will do some sort of, you know,
search on that and solve the problem.
Yeah, I mean, so even then, like-
So, um, yeah, and, and there also, I think now what these researchers
is, like, that tree s- tree search is not the load-bearing part.
That's not doing the hard thing. The-
Yeah.
... learning these internal rewards-
Yeah.
... is actually the hard part of the problem.
Yeah. Exactly.
So actually, it was more of, again, gradient descent rather than, you know,
search that cracked chess.
Tree search. Yeah, yeah. Yeah. I, I... Exactly.
I mean, it just kind of alludes a bit to my point
all about representation. Like, but then representations for what?
Like, even in the chess thing, uh, yeah, like, you, you
give, like, let's say just one bit of information
say who won. Like, you're saying, like, just that, like, you, you
just... And, yeah, I think there was even a thing where they don't even
give the rules of the game. Like, like, the model can just start with, like,
arbitrary moves and then even if they're just doing... (laughs) Yeah,
I mean, in that case, like, yeah, it just...
So, I, I think there it works because the state
comparatively smaller compared to, let's say, language or just actions in the
world. So, uh, your information, uh,
density can be, like, um, like, kind of very low and you can get away with
it. But,
uh, I, I just feel, yeah, the world is, like, much more complicated than this.
Yeah, the world is complicated. I think, yeah, there we have
very explicitly hardcoded, you know, this is the state space of the thing.
Whereas here, like, the ha- sta- state space we have
encoded is, like, way, way, way too big.
But actually the effective states which are actually important, like,
now-
No-
You know-
Yeah.
... just having a conversation with you, like, I don't have, like, a trillion
states in my head. I just have a few states, you know,
Are we not talking? Is there some emergency?
Like-
Yeah, yeah. But then-
... I need to learn all of these things rather
Yeah. The thing is... Okay, uh, I, I think there's also, like,
mutual information is needed, right?
Like (smacks lips) um, mutual information is like how much will
this... these two states, like, I wouldn't say,
like, correlated, but like if I get to know the state, like, how much
information will it re- reveal about the other state?
So in a way, I think, like our representations,
like we just have an innate
good-
Yes.
... like a base prior so that like you can just like start with
much
given. And, and I feel like that's like the,
Like it's just like different priors, uh, of like what humans have
and versus like what models have. And
even for the priors, you
like, (car hooting) like you, you, you can create better
priors only if you have access to like reality or like some
grounding signal directly. Like it's just that...
Yeah, even now, like if you can scale groundedness, I think the
problem will be cracked. And I think we just have disagreements
on what that scale of grounding
is. Like I, I agree. And I'm not de-
denying that, uh, like if you just say tomorrow like,
"Hey, we found this scalable way of grounding,"
agree. Yeah, we'll, it'll, it... the models will get really good.
I just feel like somehow solving the grounding is gonna be
tricky. Yeah.
All right.
Then I'm kind of more saying even though our right current methods for doing
grounding are really bad, gradient descent is so powerful it will
learn its internal rewards, it will learn internal states, it will-
Yeah.
Yeah, it will just learn a lot of things
use that bad gra- bad grounding and still learn things.
And it will at some point be able to even convert that bad grounding into good
grounding because it understands the field really well eventually.
Like initially it will start bad, but then you bootstrap
where even with bad grounding, it is able to learn things.
Just how human experts, you give them bad data,
something useful from that. Um,
I mean, it's not just... I think it's also about...
Yeah, tha- there's like two aspects here.
One is, like, just bad
grounding itself is
good enough. (car hooting) The other one
like, like how do you sort of
like... It, it kind of boils down to, like, the reward, like, the value
function, right? Like,
it's... Like,
uh...
How am I putting it? Like,
it's...
Yeah, I mean, I, I, I just
feel
like the initial surrogate, there'll be like a surrogate value system
humans
are, like, contributing. I mean, it, it already happens, right? Like RLHF.
There we are doing the grounding. Like, we say, "I like this,
once you do that, yeah, they just get better. So...
But still, yeah, the bottleneck was like, yeah, you had to spend like some millions
of dollars hiring people and doing it. So...
But I think from your point you're saying, like, would you have predicted
that RLHF is not needed? Like, no need to collect preference
high quality dataset? Uh, we can just, just,
uh, throw more compute and like as a gradient descent we'll just
work with noisy data itself, right?
Oh, interesting. Uh, can you make a model like-
Because that's, so that's g- the point.
... GPT 3.5 without doing RLHF?
Yeah, exactly. Like, RLHF has kind of made it usable to some
extent. Like it's like-
Yeah, it's a good question, actually.
So I... I-
Yeah, yeah. That is a good question.
Getting the answer to that will actually resolve a bit of our
Like-
Yeah, I think so.
... can I figure out a way to build GPT 3.5
Can, can pre-training... Can, can just pre-training compensate for
any... Because I see RLHF as part of-
Not just pre-training. Pre-training plus RL plus some smaller new
architecture tweaks discovered in future.
Yeah, all these are grounding. In my view, all these are grounding because new
architecture is what? Like on the space of all algorithms-
Yeah, yeah.
... human c-
All of this is grounding but, yeah, without like hiring a team of human
to do it.
Yeah, exactly. Yeah, yeah. Any, any signal that
we choose, I would say is grounding like,
and
like-
Yeah, I think I am predicting in the limit, yes,
few human experts. Like even in RLHF, like the human is not directly giving the
reward to the base model, right? The human is giving reward to a model
then giving reward back to the base model.
Yeah. Yeah.
So yeah, I am predicting in the limit we can figure
out a way such that you need a very small number of humans.
So we'll give some s- small number of rewards to this, you
know, model in the middle and that will learn, okay, this is
exactly what these humans are trying to reward and then
base model. Something like this might actually work
probability on something like this eventually working out,
Yeah, I, I also like agree because I mean, the
other data point I have is like, I don't, I don't know like how you
compare evolution. Like, evolution is like kind of embarrassingly like
simple in the sense that yeah, it just gives...
Uh, but I don't know the details. Like maybe it's very complicated, like
we, uh... like,
um, like the sexual reproduction, like that gives like genetic
diversity and stuff, but, but still like, yeah, that
didn't evolve like highly general purpose
algorithms, so... Yeah, I
mean, maybe it's
like, like... But if you
notice, right, like the-... fact of
solving, (smacks lips) like, a- again, I, I mean, the near part is just a metaphor.
Like, because, um, like, unless you solve, like, the higher level,
like, okay, this is valuable and this is not valuable, and somehow
do it in a more, more scalable way, right?
Like, um, you don't, you, you don't have to, you, you shouldn't have
those value function determination bottlenecked with
humans. Right now it's, it's very coupled.
So, so anything which decouples that and
let, like, the self-reinforcing thing
happening, um, you know, only then
I see, like, progress. And, and I, yeah, I think I need to, like, maybe make more
concrete predictions for these, um, like what I mean by
this and how ... Like, what real life events or
evidence will make me update for or against.
And, yeah, I think that will make us, uh, make it more concrete.
Like, right now I, I just have a high level view of this.
But, but yeah, I think we kind of narrowed down that.
So in your view, you're more close to the SGD itself is,
the, uh, need that comes from any form of
grounding. But I feel grounding is more important and SGD can
optimize well once the grounding is given.
And it's not like it's impossible, I just feel the bottleneck, like
once you start to rate on compute, money, energy and whatever, like, it,
like this will, this ... I, I'm not sure.
Like, uh, maybe algorithms can do it, like sam- That's why I said, like, sample
efficiency, like, uh, we, we figure it out.
But yeah, I, I ... In my view, like this is, like, the biggest
bottleneck. Like, once the other ones are saturated.
Okay. Uh, Patrick, do you have to go?
Or, because I can actually continue, but if you have to go we can (overlapping)
No, I mean, closing thoughts, like, I have to go. Yeah.
Yeah. Fair. Got it, got it. Yeah, that does make
sense. Like, I'm a- in agreement with you completely on like,
questions we need to go further.
Yeah, that's why I think I'm kind of close,
also I'm not, like, very sold on it. Yes.
I'm, I'm more ... I'm not saying narrow, but it's more like
kind of, um, it, it will
be like, uh, constant IMO goals and stuff,
and IOA goals, and we'll be, there'll be like
a slow progress, but then, um,
yeah. Like that's why like it's comparatively
But in, in a way I feel like there's, there are proto-AGI,
right? Like, by using these CLIA tools, like, they
just understand the code base, like, they're, like,
... I mean, you, you see, like, literally things, like,
them. Like, they have, like, good priors.
So I think they're kind of proto-AGI in domains
where that value function is given.
The whole point is like, can, can these values
be, um, like can we have like the
top level? Like can we, can we figure out like evolution so that
we ... Right now I think what we're doing
coming up with like the
... So, so in evolution, right, like, there is like the higher level thing
of
death, like living, and then the second lower level-
Uh, I don't want to interrupt because if you say it now I will also have to
and we will spend again 15 minutes on that.
Yeah, I just finish this analogy,
because it's ... Okay, thanks. Yeah.
So in evolution there's like a higher level of evolution itself,
like, uh, saying, okay, um,
uh, like the algorithm of evolution, like,
um, that you die and then you get to propagate
stuff, and then each environment,
uh, or maybe like each organism
does certain
optimizations. I think maybe, I think we are in a stage
where we are more like a parent trying to teach like a
kid, but
we have already gone through and then we are just teaching.
But, uh, I don't know like from there how to go to
level, where we can just like have all organisms
can, uh, adapt to any niche. Like, you just
throw this piece of bacteria in like some place and in like thousand or like
million generations it will just colonize it.
Uh, we are more like, "Okay, I, I've seen the world like
this. How can I transfer some of that to my
kid?" Uh, yeah, I think that's
my take.
Got it. Got it.
Yeah.
You, you have to say something. You can finish your thought
stop.
I mean, I ... Yeah, and, but then that will again require more replies,
problem. Like I would-
No, we can just talk and-
... like us, uh, like concrete example of a field where you
have bad grounding right now, and then I will prove to you why actually
grounding is not that bad in that particular field.
Mm-hmm.
Because like for different fields, like for like these lab experiments I have
different idea for here's how we might do grounding for, you know, language.
I have different idea, you know, persuasion, how do we correct, you know,
AI that's good at persuasion. I have different ideas like
the way we solve grounding and all of these things may be similar
as I wanted to go a bit into.
Yeah, yeah, that's my other thing, right? Like, maybe it's just, uh ...
Again, I'm not saying, uh, like even that I would agree, it's
just that again your bottleneck is like humans and the quality
of, uh, like how much high quality tokens you can
humans. Like-
Yeah, I would-
Should it be a bottleneck?
Yeah, I would rather discuss in some field why
like the number of tokens you can get from a human.
Okay.
In limit we will be able to figure out a way where it's a very small number of
tokens and you'll still cry, but I want to explain it in a specific
make more sense.
Okay. Okay. I mean, so consider
something like, um, anything which requires tacit
knowledge, right? Like, um, which is
like, it just needs
a lot of-
Can you pick, can I throw domain at them?
Hmm. Yeah, I mean, even in,
uh-Even in, like, software,
but more, like, architecture and stuff,
it's not, like
... Yeah. Ah, ah, okay, actually software might be, like,
because it's, like, easier to, like, sign in and like
stuff. But,
hmm, like,
something like, let's say,
I don't know, like
...
Yeah. Something like, um, liter-
yeah, like literature or things which are, like, more subjective.
Yeah.
Like, the models are not,
like, ah, how they will get kind of
... They can get you, like, average, I guess, and,
ah, stuff, but, again, it's bottlenecked by, "Okay,
preferences?" And-
Ah, no, no, sorry. Quick question. When you say literature, like,
Like writing a novel by any means?
Yeah, like creative writing.
Okay, fine.
Yeah, and that sort of thing.
Like, it's just-
Yeah, I think that-
... the goal is entertaining humans.
The goal is not, like, you know, technical writing for a paper and stuff?
Yeah, yeah. Not that.
Okay, good.
Yeah. Creative writing, like just, um,
writing a very, um,
things that just require, like, I don't know, a bit of
artistic
capab- not capability, but it's more like,
um
... Yeah, it just needs
... Again, I want, I want, like, place a lot
of emphasis on it because, um,
like, I, I, I also feel like the base models are,
and our service (laughs) is kind of making it worse because
it's, like, it's averaging out the thing.
So, that might kind of, like, have a nullifying
effect. Uh, so I, I'm kind
of, ah, aware of it.
But, like... Okay, even for, let's
say, STEM fields, right? Like,
like, right now, yeah, you, you've gone through all
the books and stuff. You need to hire, like, PhD, so, like,
academics and stuff. Like, how many academics are there?
Like, and so, let's say, ah, like you just, like, literally
get some level of data from all the academics.
Like, will that, is that sufficient?
Like, I, I just feel
that's still the bottleneck and it won't generalize.
Like
...
Okay. I want us to define the task a bit more
Yeah, let's say, let's say coming up with, like, better
physics experiments or something.
Yeah, like-
Sure. Yeah, yeah. Okay, that is very helpful. Yes.
Yeah. So let's say, let's do this. Like, like maybe
there is, like some, ah,
that is like, like how we have, ah,
Fields Medal or like, there are like, there may be some experimental physics,
ah, prize or something. Like, maybe, like, yeah, the
model helped some team, ah,
figure it out or something like that.
Maybe, ah, they,
like, through knowledge cutoff or something like, ah, they, they
gave it and then later they realized the humans also arrived
at the same thing the model had did or something like that.
So yeah, that's something like that.
Got it, got it. Yeah, so in that I think there are two things.
One is absorbing all the mental models that human physicists have already come up
with and we already broadly know are kind of true.
Even there you may need to do a little bit of filtering or realizing, okay, some
models here are useful, some are not useful.
Yeah.
Some might be a little bit bad.
Yeah. I, I just feel-
That there is, one is just that knowledge transfer.
Not just knowledge, but yeah, those mental models, like world models,
And then there is the more exploration of, yeah, like
searching through new world models, like what new concise explanation of, you
know, this reality can I give? What new world models can I build?
And you need grounding so that the model can do both
of these better. Ah, yeah, maybe I'll first break
that down. Like, is it learning the mo- world models of existing physic-
physicists that's hard or is it coming up with new world models
No, I think it's, ah, coming up with new stuff and yeah, like
...
Okay. Coming up with new world models is hard. Yeah, okay.
So I mean, one, I would start off with, yeah, it's not
... Like, I would not be too surprised if
literally again transformer plus RL, just how we do it right now cracks
this. And I'm also saying, look, we may need to figure out some slight
tweaks and stuff. We may need something that's not just this
But yeah, I'll first describe why I think this
Uh, so the reward signal at the end
of ... Like, you will basic- you will need some high quality data.
Like, you will need, like, a dataset of here
humanities run in the past and, you know, here are the ones that succeeded, here
are the ones that failed, here's the output we got from these
And you will have to use that as the feedback saying, okay,
experiments ... Like, just from past data of past experiments,
these experiments went well, these didn't go well, these
And you can use that as the reward signal.
And that is, yeah, a pretty
s- small dataset. Like, I know it's big, like it's thousands, but it's also not
like, you know, trillions of tokens.
Uh, like, you're not giving, you know, trillions of rewards.
You'll be giving like, you know, thousands, maybe millions of, you know,
rewards. And from that you need to learn
... Yeah, you need to learn a lot of things.
You need to learn, you know, which mental mod-
all the results we already have. You need to
learn, okay, maybe there are some data here which
Now how do I efficiently search through new world models to explain
that? Then okay, you know, what actions you might take.
Mm-hmm.
Then I have some hypotheses. Okay, here are these new world models
Now I need to propose some actual actions that actually experiment, you know, test
that hypothesis and then wait for the, you know, reward to come of that.
Like, yeah, there's a bunch of different things
but,
yeah, I feel like RL can crack a lot of these.
Uh
... Hey, how, how do I explain this?
Like
...
Actually, I have to leave now, I think.
Okay, okay. Yeah. Cool. Ah, maybe we-
Yeah.
... just have to end it here. Yeah.
Yeah. Yeah.
Yeah, well. Yeah, it was a great conversation.
Yeah, same.
I'm glad we made some progress. Yeah.
Yeah.