let's talk about a failure of intuition
that many of us suffer from it's really
a failure to detect a certain kind of
danger I'm gonna describe a scenario
that I think is both terrifying and
likely to occur and that's not a good
combination as it turns out and yet
rather than be scared most of you will
feel that what I'm talking about is kind
of cool
I'm gonna describe how the gains we make
in artificial intelligence could
ultimately destroy us in fact I think
it's very difficult to see how they
won't destroy us or inspire us to
destroy ourselves and yet if you're
anything like me you'll find that it's
fun to think about these things and that
that response is part of the problem
okay that response should worry you if I
were to convince you in this talk that
we were likely to suffer a global famine
either because of climate change or some
other catastrophe and that your
grandchildren or their grandchildren are
very likely to live like this
you wouldn't think interesting I like
this TED talk
famine isn't fun death by science
fiction on the other hand is fun and one
of the things that worries me most about
the development of AI at this point is
that we seem unable to marshal an
appropriate emotional response to the
dangers that lie ahead I'm unable to
marshal this response and I'm giving
this talk it's as though we stand before
two doors behind door number one we stop
making progress in building intelligent
machines our computer hardware and
software just stops getting better for
some reason now take a moment to
consider why this might happen but given
how valuable intelligence and automation
are we will continue to improve our
technology if we are at all able to what
could stop us from doing this a
full-scale nuclear war a global pandemic
an asteroid impact Justin Bieber
becoming president United States the
point is something would have to destroy
civilization as we know it hey you have
to imagine how bad it would have to be
to prevent us from making improvements
in our technology permanently a
generation after generation almost by
definition this is the worst thing
that's ever happened in human history so
the only alternative and this is what
lies behind door number two is that we
continue to improve our intelligent
machines year after year after year and
a certain point we will build machines
that are smarter than we are
and once we have machines that are
smarter than we are they will begin to
improve themselves and then we risk with
the mathematician AI J good called an
intelligence explosion that the process
could get away from us now this is often
caricatured as I have here as a fear
that armies of malicious robots will
attack us but that isn't the most likely
scenario a it's not that our machines
will become spontaneously malevolent
because concern is really that we will
build machines that are so much more
competent than we are that the slightest
divergence between their goals in our
own could destroy us just think about
how we relate to ants okay we don't hate
them we don't go out of our way to harm
them in fact sometimes we take pains not
to harm them we just we step over them
on the sidewalk but whenever their
presence
seriously conflicts with one of our
goals let's say when constructing a
building like this one we annihilate
them without a qualm the concern is that
we will one day build machines that
whether they're conscious or not could
treat us with similar disregard now I
suspect this seems far-fetched to many
of you I bet they're those of you who
doubt that super intelligent AI is
possible
much less inevitable right but then you
must find something wrong with one of
the following assumptions and they're
only three of them intelligence is a
matter of
information processing in physical
systems okay actually this is a little
bit more than an assumption but we have
already built narrow intelligence into
our machines and many of these machines
perform at a level of superhuman
intelligence already and we know that
mere matter can give rise to what is
called general intelligence and ability
to think flexibly across multiple
domains because our brains have managed
it right maybe there's just atoms in
here as long as we continue to build
systems of atoms that display more and
more intelligent behavior we will
eventually unless we are interrupted we
will eventually build general
intelligence into our machines it's
crucial to realize that that the rate of
progress doesn't matter if any progress
is enough to get us into the end zone we
don't need Moore's law to continue we
don't we don't need exponential progress
we just need to keep going the second
assumption is that we will keep going we
will continue to improve our intelligent
machines and given the value of
intelligence
you know intelligence is either the
source of everything we value or we need
it to safeguard everything we value it
is our most valuable resource so we want
to do this we have problems that we
desperately need to solve we want to
cure diseases like Alzheimer's and
cancer you want to understand economic
systems we want to improve our climate
science so we will do this if we can the
Train is already out of the station and
there's no brake to pull finally we
don't stand on a peak of intelligence or
anywhere near it likely and this really
is the crucial insight this is what
makes our situation so precarious and
this is what what makes our intuitions
about risk so unreliable now just
consider the smartest person who's ever
lived and almost everyone's shortlist
here is John von Neumann maybe the
impression that fun women made on the
people around him
and this included the greatest
mathematicians and physicists of his
time it's fairly well documented
if if only half the stories about him
are half true that no question he's one
of the smartest people who's ever lived
so consider the spectrum of intelligence
here we have john von neumann and then
we have you and me and then we have a
chicken sorry a chicken it's no reason
for me to make this talk more depressing
that needs to be it seems overwhelmingly
likely however that the spectrum of
intelligence extends much further than
we currently conceive and if we build
machines that are more intelligent than
we are they will very likely explore
this spectrum in ways that we can't
imagine and exceed us in ways that we
can't imagine and it's important to
recognize that this is true by virtue of
speed alone
right so imagine we just built a super
intelligent AI right that was no smarter
than your average team of researchers at
Stanford or MIT well electronic circuits
function about a million times faster
than biochemical ones okay so this
machine should think about a million
times faster than the minds have built
it so you said it running for a week and
it will perform 20,000 years of human
level intellectual work week after week
after week how could we even understand
much less constrain in mind making this
sort of progress the other thing that's
worrying frankly is that imagine that
imagine the best-case scenario so
imagine we we hit upon a design of super
intelligent AI that has no safety
concerns we have the perfect design the
first time around it's as though we've
been handed an oracle that behaves
exactly as intended well this machine
would be the perfect labor-saving device
it can design the machine that can build
the machine that can do any physical
work powered by sunlight more or less
for the cost of raw materials okay so
we're talking about the end of human
drudgery we're also talking about the
end of most intellectual work so what
would a piece like ourselves do in this
our
stands well we'd be free to play frisbee
and give each other massages add some
LSD and some questionable wardrobe
choices and the whole world could be
like Burning Man now that might sound
pretty good but ask yourself what would
happen under our current economic and
political order it seems likely that we
would witness a level of wealth
inequality and unemployment that we have
never seen before absent the willingness
to immediately put this new wealth to
the surface of all humanity okay well a
few trillion areas could raise the
covers of our business magazines while
the rest of the world would be free to
starve and what would the Russians or
the Chinese do if they heard that some
company in Silicon Valley was about to
deploy a super intelligent AI this
machine would be capable of waging war
right right whether to rest rail or
cyber with unprecedented power this is a
winner-take-all scenario dude to be six
months ahead of the competition here is
to be 500 thousand years ahead
at a minimum okay so it seems that even
mere rumors of this kind of breakthrough
could cause our species to go berserk
now one of the the most frightening
things in my view at this moment are the
kinds of things that AI researchers say
when they want to be reassuring and the
most common reason we're told not to
worry is time this is all a long way off
don't you know this is this is probably
fifty or a hundred years away one
researcher has said worrying about AI
safety is like worrying about
overpopulation on Mars this is the
Silicon Valley version of don't worry
your pretty little head about it
no one seems to notice that referencing
the time horizon is a total non sequitur
okay if intelligence is just a matter of
information processing and we continue
to improve our machines we will produce
some form of super intelligence and we
have no idea how long it will take us to
create the conditions to do that safely
let me say that again we have no idea
how long it will take us to create the
conditions to do that safely if you
haven't noticed 50 years is not what it
used to be and this is 50 years in
months
this is how long we've had the iPhone
this is how long with Simpsons has been
on television fifty years is not that
much time to meet one of the greatest
challenges our species will ever face
and once again we seem to be failing to
have an appropriate emotional response
to what we have every reason to believe
is coming.the the computer scientist
Stuart Russell has a nice analogy here
he said imagine that we received a
message from an alien civilization which
read people of Earth we will arrive on
your planet in 50 years get ready and
now we're just counting down the months
until the mothership lands okay we would
feel a little more urgency than we do
another reason we're told not to worry
is that these machines can't help but
share our values because they will be
literally extensions of ourselves
they'll be grafted on to our brains and
will essentially become their limbic
systems well take a moment to consider
that the safest and only prudent path
forward recommended is to implant this
technology directly into our brains now
that this mat may in fact be the safest
and only prudent path forward but
usually one safety concerns about a
technology have to be pretty much worked
out before you stick it inside your head
the deeper problem is that building
super intelligent AI on its own seems
likely to be easier than building super
intelligent AI and having a completed
neuroscience that allows us to
seamlessly integrate our minds
and given that the companies and
governments doing this work are likely
to perceive themselves to be in a race
against all others given that to win
this race is to win the world provide a
you don't destroy it in the next moment
then it seems likely that whatever is
easier to do will get done first now
unfortunately I have a solution to this
problem or from recommending that more
of us think about it I think we need
something like a Manhattan Project on
the topic of artificial intelligence not
to build it because I think we'll
inevitably do that but to understand how
to avoid an arms race and to build it in
a way that is aligned with our interests
when you're talking about super
intelligent AI that can make changes to
itself it would seem it seems that we
only have one chance to get the initial
conditions right and even then we will
need to absorb the economic and
political consequences of getting them
right but the moment we admit that
information processing is the source of
intelligence that some appropriate
computational system is what the basis
of intelligence is and we admit that we
will improve these systems continuously
and we admit that the horizon of
cognition very likely far exceeds what
we currently know then we have to admit
that we're in the process of building
some sort of God now would be a good
time to make sure it's a god we can live
with thank you very much
[Applause]