... perceives any danger. Can you just
roughly explain what you think the dangers might be?
Yeah, so the- the- the- the danger, uh, re- really, AI
is, um,
perhaps, uh,
more dangerous than, say, mismanaged, uh, aircraft design
or production maintenance or- or- or b- bad car production, uh, in the sense that it is,
it has the potential, uh, however small one may regard that probability, but it is
non-trivial. It has the potential of civilizational destruction.
Regulations are really only put into effect after something terrible has happened.
That's correct.
If that's the case for AI, and we only put regulations after something terrible has
happened, it may be too late to actually put the regulations in place.
The AI may be in control at that point.
You think
that's real? It is-
it is conceivable that AI could take control and reach a point where you couldn't turn it
off and it would be making-
making the decisions for people?
Yeah, absolutely.
Absolutely?
No, it's that's the- that's definitely the- where things are headed.
Uh, um, I- I guess I sort of agree with Geoff Hinton that maybe it's a 10 to 20%
chance of annihilation. Uh, but look on the bright side.
That's 80 to 90% probability of a great outcome.
And then furthermore, there are concerns around if an AGI ever
becomes very, very powerful, which is possible,
maybe it will want to go rogue, being that it is an agent.
So this is a concern that exists with this unprecedented, not yet existing
technology.
It's not that it's going to actively hate humans and want to harm them.
But it's just going to be too powerful.
And I think a good analogy would be the way humans treat animals.
It's not that we hate animals. I think humans love animals and have a lot of affection
for them,
but
when the time comes to build a highway between two cities,
we are not asking the animals for permission.
We just do it because it's important for us.
And I think by default, that's the kind of relationship that's going to be between us
and AGIs which are truly autonomous and operating on their own
behalf.
Some folks who
consider all the different problems with a super intelligent AI system.
So,
uh, one of them is Eliezer Yudkowsky.
He warns that AI will likely kill all humans.
Um, to what degree do you
disagree with
that trajectory?
So first of all, I will say I think that
there's some chance of that and it's really important to acknowledge it because if we
don't talk about it, if we don't treat it as potentially real, we won't put enough effort
into solving it. And I think we do have to discover new techniques
to be able to solve it.
Are you afraid of losing control of the AGI itself?
There's a lot of, uh, people who worry about existential risk, not because of state
actors, not because of security concerns, but because of the AI itself.
That is not my top worry as I currently see things.
There have been times I worried about that more.
There may be times again in the future where that's my
top worry. It's not my top worry right now.
What's your intu-
You know, I- I think I've- I think I've often said that, you know, my- my chance
that something goes, you know, really quite catastrophically wrong on the
scale of- of, you know, human civilization, you know, might be somewhere between 10 and
25% when you put together the risk of something going wrong
with the model itself
with, you know, something going wrong with human, you know, people or organizations or
nation-states misusing the model or- or it kind of inducing conflict
among them or- or just some way in which kind of society can't- can't handle it.
Significantly. I- I do think there are a few things.
One is this risk of the- the AI models autonomously acting
in ways that are- are not in line with- with human interest, right?
If you have a country of geniuses in a data center, a natural question, how could you not
ask this question, "Well, what is their intent?
What do they plan to do?" You would certainly ask, "Well, is someone controlling them?
Are they acting on someone's behalf?" But you would also ask, "Well, what is their
intent?" And because we grow these systems, we don't train them, I don't think it's safe
to assume they'll do exactly what their human designers or users
want them to do. So I think there's- there's real risk of that.
I think it could be a threat to- to all- to, you know, to- to- to kind of all of humanity
and, like,
seriously.
Paint a scenario of what going bad could look like.
Well, it's- it's hard to do because you're talking about systems that potentially
have, you know, superhuman intelligence, right?
So there are- there are many things which- and ways in which things could go bad in the
world. Um, people sometimes point to, I don't know, engineered pathogens, right?
Maybe the- you know, a super intelligence could design an engineered pathogen.
It could be much more mundane things.
Um, maybe with AGI, you know, it gets used to
destabilize, uh, democracy in the world, um, with, you know, propaganda or all sorts
of other things like that.
Ridiculous question. What's your PDOOM?
Probability that human civilization destroys itself.
Um, what I would say is it's definitely non-zero and it's probably
non-neg- negligible. So that in itself is pretty sobering, and
my- my view is it's just hugely uncertain, right? What these techs are.
Uh, could you paint the worst case scenario for me?
Sure. (laughs) Well, look. Worst case I think has been covered a lot in science fiction.
What is a path forward that ensures
safety?
I don't know. I- I can't see a path that guarantees
safety.... that we're entering a period of great uncertainty where we're dealing
with things we've never dealt with before.
And normally, the first time you deal with something totally novel, you get it wrong.
And we can't afford to get it wrong with these things. Can't afford to get it wrong, why?
Well, because they might take over.
Take over from humanity? Yes, that's a possibility. Why would they want to?
I'm not saying it will happen.
If we could stop them ever wanting to, that would be great.
But it's not clear we can stop them ever want-
If you, um, if you had a closing message for, for my, for my listeners about AI and AI
safety,
what would that be, Geoffrey?
There's still a chance that we can figure out how to develop AI
that won't want to take over. And
we have to face the possibility
that unless we do something
soon,
we're near the end.
So, we are on a trajectory to build machines that
are smarter and smarter.
And one day, it's very plausible that they will be smarter than us.
And then
they will have their own agency,
their own goals,
which may not be aligned with ours.
What happens to us then?
Poof,
we are
blindly driving into a fog.
Despite the warnings of scientists like myself that this trajectory could
lead to loss of control,
beside me in the car are my children, my grandson,
my loved ones.
Who is beside you
in the car?
Who is in your care for the future? Regulation.
And, and, and just wrap it up, you know, in very clear terms, what is
the worst case scenario that you're worried about?
Well, the worst case scenario is, uh, human
extinction.
I'm worried about the future AIs. I'm worried about the AI that is good
enough at AI research to build the AI that builds the AI that is smarter than us and
kills everyone. The AI gets the universe it wants most, that is a universe that does
not happen to have us in it, and that is how indifference kills you.
If we could just get 50 tries at building superintelligences, you know, it could be like,
"Our clever alignment theory didn't work, it killed everyone. Let's build another one.
Oh, that one killed everyone too. Wow, the second crazy theory we had didn't work
either." The basic description I would give to the current scenario is, if anyone builds
it, everyone dies.
...
of the loop. Okay. So, okay, so even if we have a virtual society, we need some
factories and some hardware, but we don't need the humans in the loop.
And indeed, they are not very useful.
Okay, so, um, this is how we can
build this singularity. Um,
so now first look at an outward explosion.
So, um, we build more and more computers, and they fill up more and more space.
And, well, at some point, there is no space anymore for humans
(laughs) um,
to observe
the entire world, you know? They will be swallowed. Yeah.
And, I mean, you could, you know, send it to uni- universe.
So, either, um, you get into the singularity, get sucked into it, that would be probably
the most positive scenario, or, um, you get killed.
So, if you want to see a nice movie about computers taking over, I mean, Terminator is
great for the special effects. Yeah, but story-wise, there's a very old movie, Colossus,
which is much more realistic. You know, no special effects, very softly boring, but
I think it's much more realistic.
But I think the development of full artificial
intelligence could spell the end of the human race.
Once humans develop artificial intelligence, it would take off on
its own and redesign itself at an ever-increasing rate.
Shoo-ba-dee-doo. Ba-ba, ba-ba, ba-ba, bop, bop,
bop. Ba-ba-bop, bop, bop, ba-ba-bop, bop, bop. Shoo-ba-dee-doo.
Bop, bop, ba-ba, bop, bop, bop, bop, bop, bop, bop, bop, bop,
shoo-ba-dee-doo,
shoo-ba-dee-doo, shoo-ba-dee, bop, bop, bop, bop, bop, bop, bop, bop.