WEBVTT
00:05.099 --> 00:07.419
... perceives any danger. Can you just
00:08.059 --> 00:10.019
roughly explain what you think the dangers might
be?
00:10.019 --> 00:14.880
Yeah, so the- the- the- the danger, uh, re-
really, AI
00:14.960 --> 00:15.800
is, um,
00:17.719 --> 00:18.739
perhaps, uh,
00:20.639 --> 00:25.600
more dangerous than, say, mismanaged, uh, aircraft
design
00:25.799 --> 00:30.280
or production maintenance or- or- or b- bad car
production, uh, in the sense that it is,
00:31.099 --> 00:34.899
it has the potential, uh, however small one may
regard that probability, but it is
00:35.399 --> 00:39.319
non-trivial. It has the potential of
civilizational destruction.
00:40.180 --> 00:43.520
Regulations are really only put into effect after
something terrible has happened.
00:43.520 --> 00:44.299
That's correct.
00:44.299 --> 00:48.220
If that's the case for AI, and we only put
regulations after something terrible has
00:48.239 --> 00:51.459
happened, it may be too late to actually put the
regulations in place.
00:51.459 --> 00:53.239
The AI may be in control at that point.
00:53.399 --> 00:54.619
You think
00:55.899 --> 00:57.020
that's real? It is-
00:57.680 --> 01:01.180
it is conceivable that AI could take control and
reach a point where you couldn't turn it
01:01.180 --> 01:02.139
off and it would be making-
01:03.139 --> 01:04.499
making the decisions for people?
01:04.599 --> 01:06.019
Yeah, absolutely.
01:06.559 --> 01:07.279
Absolutely?
01:07.639 --> 01:09.999
No, it's that's the- that's definitely the- where
things are headed.
01:10.879 --> 01:15.599
Uh, um, I- I guess I sort of agree with Geoff
Hinton that maybe it's a 10 to 20%
01:15.620 --> 01:18.339
chance of annihilation. Uh, but look on the bright
side.
01:18.339 --> 01:21.019
That's 80 to 90% probability of a great outcome.
01:24.739 --> 01:29.579
And then furthermore, there are concerns around if
an AGI ever
01:29.639 --> 01:33.099
becomes very, very powerful, which is possible,
01:33.759 --> 01:36.900
maybe it will want to go rogue, being that it is
an agent.
01:37.479 --> 01:42.400
So this is a concern that exists with this
unprecedented, not yet existing
01:42.439 --> 01:42.999
technology.
01:44.019 --> 01:48.679
It's not that it's going to actively hate humans
and want to harm them.
01:50.959 --> 01:52.620
But it's just going to be too powerful.
01:53.560 --> 01:57.480
And I think a good analogy would be the way humans
treat animals.
01:59.139 --> 02:02.819
It's not that we hate animals. I think humans love
animals and have a lot of affection
02:02.880 --> 02:03.200
for them,
02:04.560 --> 02:04.859
but
02:05.640 --> 02:08.759
when the time comes to build a highway between two
cities,
02:09.759 --> 02:11.479
we are not asking the animals for permission.
02:12.279 --> 02:14.979
We just do it because it's important for us.
02:16.439 --> 02:20.660
And I think by default, that's the kind of
relationship that's going to be between us
02:21.220 --> 02:26.099
and AGIs which are truly autonomous and operating
on their own
02:26.139 --> 02:26.759
behalf.
02:29.299 --> 02:31.479
Some folks who
02:32.219 --> 02:35.839
consider all the different problems with a super
intelligent AI system.
02:35.839 --> 02:36.099
So,
02:36.999 --> 02:39.340
uh, one of them is Eliezer Yudkowsky.
02:40.560 --> 02:43.959
He warns that AI will likely kill all humans.
02:43.999 --> 02:45.359
Um, to what degree do you
02:46.220 --> 02:47.159
disagree with
02:47.799 --> 02:48.660
that trajectory?
02:50.200 --> 02:52.060
So first of all, I will say I think that
02:53.419 --> 02:56.939
there's some chance of that and it's really
important to acknowledge it because if we
02:56.959 --> 03:00.619
don't talk about it, if we don't treat it as
potentially real, we won't put enough effort
03:00.639 --> 03:05.299
into solving it. And I think we do have to
discover new techniques
03:05.999 --> 03:07.059
to be able to solve it.
03:08.179 --> 03:11.300
Are you afraid of losing control of the AGI
itself?
03:11.300 --> 03:15.619
There's a lot of, uh, people who worry about
existential risk, not because of state
03:15.659 --> 03:18.999
actors, not because of security concerns, but
because of the AI itself.
03:18.999 --> 03:21.659
That is not my top worry as I currently see
things.
03:21.659 --> 03:23.020
There have been times I worried about that more.
03:23.020 --> 03:24.739
There may be times again in the future where
that's my
03:25.399 --> 03:27.499
top worry. It's not my top worry right now.
03:27.499 --> 03:28.099
What's your intu-
03:30.920 --> 03:35.739
You know, I- I think I've- I think I've often said
that, you know, my- my chance
03:35.760 --> 03:40.639
that something goes, you know, really quite
catastrophically wrong on the
03:40.639 --> 03:45.079
scale of- of, you know, human civilization, you
know, might be somewhere between 10 and
03:45.120 --> 03:49.999
25% when you put together the risk of something
going wrong
03:50.059 --> 03:51.179
with the model itself
03:51.780 --> 03:56.579
with, you know, something going wrong with human,
you know, people or organizations or
03:56.719 --> 04:01.399
nation-states misusing the model or- or it kind of
inducing conflict
04:01.539 --> 04:06.059
among them or- or just some way in which kind of
society can't- can't handle it.
04:06.059 --> 04:09.099
Significantly. I- I do think there are a few
things.
04:09.099 --> 04:13.999
One is this risk of the- the AI models
autonomously acting
04:14.019 --> 04:17.099
in ways that are- are not in line with- with human
interest, right?
04:17.099 --> 04:21.319
If you have a country of geniuses in a data
center, a natural question, how could you not
04:21.379 --> 04:23.459
ask this question, "Well, what is their intent?
04:23.459 --> 04:26.979
What do they plan to do?" You would certainly ask,
"Well, is someone controlling them?
04:26.979 --> 04:30.539
Are they acting on someone's behalf?" But you
would also ask, "Well, what is their
04:30.559 --> 04:35.179
intent?" And because we grow these systems, we
don't train them, I don't think it's safe
04:35.239 --> 04:39.799
to assume they'll do exactly what their human
designers or users
04:40.139 --> 04:43.159
want them to do. So I think there's- there's real
risk of that.
04:43.159 --> 04:47.219
I think it could be a threat to- to all- to, you
know, to- to- to kind of all of humanity
04:47.259 --> 04:47.739
and, like,
04:50.999 --> 04:51.659
seriously.
04:51.659 --> 04:54.419
Paint a scenario of what going bad could look
like.
04:54.419 --> 04:59.119
Well, it's- it's hard to do because you're talking
about systems that potentially
04:59.219 --> 05:01.559
have, you know, superhuman intelligence, right?
05:01.839 --> 05:05.339
So there are- there are many things which- and
ways in which things could go bad in the
05:05.359 --> 05:09.579
world. Um, people sometimes point to, I don't
know, engineered pathogens, right?
05:09.599 --> 05:13.079
Maybe the- you know, a super intelligence could
design an engineered pathogen.
05:13.339 --> 05:15.059
It could be much more mundane things.
05:15.079 --> 05:19.259
Um, maybe with AGI, you know, it gets used to
05:19.419 --> 05:24.399
destabilize, uh, democracy in the world, um, with,
you know, propaganda or all sorts
05:24.419 --> 05:25.239
of other things like that.
05:27.319 --> 05:29.919
Ridiculous question. What's your PDOOM?
05:29.979 --> 05:32.599
Probability that human civilization destroys
itself.
05:32.599 --> 05:37.219
Um, what I would say is it's definitely non-zero
and it's probably
05:37.259 --> 05:42.139
non-neg- negligible. So that in itself is pretty
sobering, and
05:42.339 --> 05:46.399
my- my view is it's just hugely uncertain, right?
What these techs are.
05:46.439 --> 05:48.779
Uh, could you paint the worst case scenario for
me?
05:48.979 --> 05:53.339
Sure. (laughs) Well, look. Worst case I think has
been covered a lot in science fiction.
05:55.099 --> 05:59.919
What is a path forward that ensures
05:59.939 --> 06:00.399
safety?
06:00.399 --> 06:04.899
I don't know. I- I can't see a path that
guarantees
06:04.919 --> 06:09.847
safety.... that we're entering a period of great
uncertainty where we're dealing
06:09.847 --> 06:11.867
with things we've never dealt with before.
06:12.907 --> 06:16.268
And normally, the first time you deal with
something totally novel, you get it wrong.
06:16.828 --> 06:20.587
And we can't afford to get it wrong with these
things. Can't afford to get it wrong, why?
06:20.587 --> 06:22.627
Well, because they might take over.
06:23.627 --> 06:26.907
Take over from humanity? Yes, that's a
possibility. Why would they want to?
06:26.907 --> 06:28.067
I'm not saying it will happen.
06:29.107 --> 06:31.587
If we could stop them ever wanting to, that would
be great.
06:32.088 --> 06:34.108
But it's not clear we can stop them ever want-
06:34.187 --> 06:38.828
If you, um, if you had a closing message for, for
my, for my listeners about AI and AI
06:38.907 --> 06:39.327
safety,
06:40.047 --> 06:40.948
what would that be, Geoffrey?
06:42.387 --> 06:46.307
There's still a chance that we can figure out how
to develop AI
06:46.927 --> 06:48.647
that won't want to take over. And
06:50.028 --> 06:51.708
we have to face the possibility
06:52.707 --> 06:54.267
that unless we do something
06:55.287 --> 06:55.847
soon,
06:56.967 --> 06:58.027
we're near the end.
07:01.508 --> 07:06.467
So, we are on a trajectory to build machines that
07:06.467 --> 07:07.827
are smarter and smarter.
07:08.867 --> 07:12.567
And one day, it's very plausible that they will be
smarter than us.
07:13.207 --> 07:13.567
And then
07:14.227 --> 07:15.807
they will have their own agency,
07:18.027 --> 07:19.047
their own goals,
07:20.627 --> 07:23.727
which may not be aligned with ours.
07:25.427 --> 07:27.047
What happens to us then?
07:28.947 --> 07:29.487
Poof,
07:31.227 --> 07:32.207
we are
07:32.847 --> 07:35.007
blindly driving into a fog.
07:35.967 --> 07:40.807
Despite the warnings of scientists like myself
that this trajectory could
07:40.867 --> 07:43.007
lead to loss of control,
07:44.027 --> 07:48.807
beside me in the car are my children, my grandson,
07:49.067 --> 07:49.967
my loved ones.
07:51.987 --> 07:53.607
Who is beside you
07:54.247 --> 07:54.927
in the car?
07:56.767 --> 08:00.707
Who is in your care for the future? Regulation.
08:00.707 --> 08:05.567
And, and, and just wrap it up, you know, in very
clear terms, what is
08:05.627 --> 08:07.367
the worst case scenario that you're worried about?
08:07.367 --> 08:12.087
Well, the worst case scenario is, uh, human
08:12.127 --> 08:12.747
extinction.
08:15.067 --> 08:19.807
I'm worried about the future AIs. I'm worried
about the AI that is good
08:19.827 --> 08:24.107
enough at AI research to build the AI that builds
the AI that is smarter than us and
08:24.107 --> 08:29.087
kills everyone. The AI gets the universe it wants
most, that is a universe that does
08:29.127 --> 08:32.567
not happen to have us in it, and that is how
indifference kills you.
08:32.787 --> 08:37.047
If we could just get 50 tries at building
superintelligences, you know, it could be like,
08:37.047 --> 08:41.027
"Our clever alignment theory didn't work, it
killed everyone. Let's build another one.
08:41.047 --> 08:45.267
Oh, that one killed everyone too. Wow, the second
crazy theory we had didn't work
08:45.267 --> 08:49.247
either." The basic description I would give to the
current scenario is, if anyone builds
08:49.267 --> 08:50.627
it, everyone dies.
08:50.627 --> 08:50.827
...
08:54.067 --> 08:58.787
of the loop. Okay. So, okay, so even if we have a
virtual society, we need some
08:59.067 --> 09:02.407
factories and some hardware, but we don't need the
humans in the loop.
09:02.807 --> 09:04.627
And indeed, they are not very useful.
09:06.627 --> 09:09.167
Okay, so, um, this is how we can
09:09.987 --> 09:12.427
build this singularity. Um,
09:13.087 --> 09:15.607
so now first look at an outward explosion.
09:16.107 --> 09:19.947
So, um, we build more and more computers, and they
fill up more and more space.
09:21.067 --> 09:25.187
And, well, at some point, there is no space
anymore for humans
09:26.167 --> 09:26.567
(laughs) um,
09:27.187 --> 09:27.947
to observe
09:28.947 --> 09:31.367
the entire world, you know? They will be
swallowed. Yeah.
09:31.907 --> 09:34.487
And, I mean, you could, you know, send it to uni-
universe.
09:35.207 --> 09:39.847
So, either, um, you get into the singularity, get
sucked into it, that would be probably
09:39.867 --> 09:43.047
the most positive scenario, or, um, you get
killed.
09:44.187 --> 09:48.667
So, if you want to see a nice movie about
computers taking over, I mean, Terminator is
09:48.687 --> 09:53.547
great for the special effects. Yeah, but
story-wise, there's a very old movie, Colossus,
09:53.927 --> 09:58.847
which is much more realistic. You know, no special
effects, very softly boring, but
09:59.207 --> 10:01.067
I think it's much more realistic.
10:03.987 --> 10:08.387
But I think the development of full artificial
10:08.427 --> 10:12.207
intelligence could spell the end of the human
race.
10:15.927 --> 10:20.727
Once humans develop artificial intelligence, it
would take off on
10:20.727 --> 10:24.667
its own and redesign itself at an ever-increasing
rate.
10:24.667 --> 10:29.207
Shoo-ba-dee-doo. Ba-ba, ba-ba, ba-ba, bop, bop,
10:29.307 --> 10:33.887
bop. Ba-ba-bop, bop, bop, ba-ba-bop, bop, bop.
Shoo-ba-dee-doo.
10:33.887 --> 10:34.647
Bop, bop, ba-ba, bop, bop, bop, bop, bop, bop,
bop, bop, bop,
10:35.447 --> 10:36.547
shoo-ba-dee-doo,
10:37.447 --> 10:40.047
shoo-ba-dee-doo, shoo-ba-dee, bop, bop, bop, bop,
bop, bop, bop, bop.
Enter "Subscribe: " followed by Signal username, email address or whatsapp phone number. You will receive atmost one update per month. View previous updates here.
Alternatively, you can use RSS, or a third-party service like blogtrottr.
Enter "Comment: " followed by comment. Alternatively, contact me