I'm James I'm a writer an artist and I
make work about technology I do things
like draw life-sized outlines of
military drones in city streets around
the world so that people can start to
think and get their heads around these
really quite hard to see and hard to
think about technologies I make things
like neural networks that predict the
results of elections based on weather
reports because I'm intrigued about what
the actual possibilities of these kind
of weird new technologies are last year
I built my own self-driving car but
because I don't really trust technology
I also designed a trap for it and I do
these things mostly because I find them
completely fascinating but also because
I think when we talk about technology
we're kind of largely talking about
ourselves and the way that we understand
the world so here's a story about
technology this is a surprise egg video
right it's basically a video of someone
opening up loads of chocolate eggs and
showing the toys inside to the viewer
that's it that's all it does for seven
long minutes and I want you to notice
two things about this first of all this
video has 30 million views all right and
the other thing is that it comes from a
channel that has 6.3 million subscribers
that has a total of 8 billion views and
it's all just more videos like this 30
million people watching a guy opening up
these eggs
it sounds pretty weird but if you search
for surprise eggs on YouTube it'll tell
you there's 10 million of these videos
and I think that's an undercount right I
think there's way way more of these if
you keep searching they're endless
there's millions and millions of these
videos in increasingly Barok
combinations of brands and materials and
there's more and more of them being
uploaded every single day like this this
is a strange world right but the thing
is it's not adults who are watching
these videos it's kids small small
children these videos are like cracks
for little kids there's something about
like the repetition
the constant little dopamine hit of the
reveal that completely hooks the men and
little kids watched these videos over
and over and over again and they do it
for hours and hours and hours and if you
try and take the screen away from them
they'll scream and scream and scream and
if you don't believe me and I've already
seen people in the audience nodding if
you don't believe me find someone with
small children and ask them and they'll
know about the surprise egg videos right
so this is this is where we start its
2018 and someone or lots of people are
using the same mechanism that like
Facebook and Instagram were using to get
you to keep checking that out and
they're using it on YouTube to hack the
brains of very small children in return
for advertising revenue at least I hope
that's what they're doing right I hope
that's what they're doing it for because
there's there's easier ways of making ad
revenue on on YouTube right you can just
make stuff up or steal stuff so if you
search for like really popular kids
cartoons like Peppa Pig or paw Patrol
you'll find that there's millions and
millions of these online as well of
course most of them aren't posted by the
original content creators they come from
loads and loads of different kind of
random accounts and it's impossible to
know who's posting them or what their
motives might be right and does that
sound kind of familiar because really
it's exactly the same mechanism that's
happening across most of our digital
services where it's impossible to know
where this information is coming from
it's basically fake news for kids and
we're training them from birth to click
on the very first link that comes along
regardless of what the source is that
doesn't seem like a terribly good idea
here's another thing that's really
really big on kids YouTube this is
called the finger family song again I
just heard someone groan in the audience
this is the finger family song right
this is the very first one I could find
it's from 2007 and it only has 200,000
views which is like nothing in this game
right but it has this insanely ear wormy
tune which I'm not going to play to you
because it will sear itself into your
brain in the same way that it's seared
itself into mine and I'm not going to do
that to you but like the surprise eggs
like it's gone inside kids heads and
like addicted them to it so within a few
years these finger family videos start
appearing everywhere and you get
versions in different
which is with popular kids cartoons
using food or frankly using whatever
kind of animation elements you seem to
have lying around okay and once again
there are millions and millions and
millions of these videos available
online in all of these kind of insane
combinations and the more time you start
to spend with them the crazier and
crazier you start to feel that you might
be and that's the thing I find that's
that's where I kind of launched into
this that feeling of deep strangeness
and deep lack of understanding of how
this thing was constructed that seems to
be presented around me right because
it's impossible to know where these
things are coming from like who is
making them some of them appear to be
made of teams of professional animators
some of them are just randomly assembled
by software some of them are you know
quite wholesome looking young kids
entertainers and some of them are from
people who really clearly shouldn't be
around children at all okay and and once
again this this impossibility of
figuring out who's making this stuff
like is this a bot is this a person is
this a troll what does it mean that we
can't tell the difference between these
things anymore and again doesn't that
uncertainty feel kind of familiar right
now so the main way people get views on
their videos and remember views mean
money is that they stuff the titles of
these videos with with these popular
terms so you take like surprise eggs and
then you add poor patrol or easter egg
or whatever these things are all of
these words from other popular videos
into your title until you end up with
this kind of meaningless mash of
language right that doesn't make sense
to humans at all
because of course it's only like really
tiny kids who are watching your video
and what the hell do they know like
you're real audience for this stuff is
is is software is the algorithms it's
the software that YouTube uses to select
which videos are like other videos to
make them popular to make them recommend
it and that's why you end up with this
kind of completely meaningless mash most
of title and of content but the thing is
you have to remember there really are
still people within this algorithmically
optimized system people who are kind of
increasingly forced to act out these
increasingly bizarre combinations of
words like
kind of desperate improvisation artists
responding to their kind of combined
screams of a million toddlers at once I
though real people traps within these
systems and that's the other deeply
strange thing about this algorithmically
driven culture is even if you're human
you have to end up behaving like a
machine just to survive and also on the
other side of the screen there still are
these little kids watching this stuff
right stuck at their full attention
grabbed by these weird mechanisms and
most of these kids are too small to even
use a website right there just kind of
hammering on the screen with a little
hands alright and so there's also play
where it just keeps playing these videos
over and over and over in a loop
endlessly for hours and hours at a time
and there's so much weirdness in the
system now
the autoplay takes you to some pretty
strange places this is how within like a
dozen steps you can go from a cute video
of a counting train to masturbating
Mickey Mouse yeah
I'm sorry about that this does get worse
this is what happens when all of these
different keywords or these different
kind of pieces of attention this
desperate generation of content all
comes together into a single place this
is where all those deeply weird keywords
come home to roost you crossbreed the
finger family video with some live
action superhero stuff you add in some
like weird trollish in jokes or
something and suddenly you come to a
very weird place indeed the stuff that
tends to upset parents is the stuff that
has kind of violent or sexual content
right children's cartoons getting
getting assaulted getting killed and
weird pranks that actually genuinely
terrified children or you have its
software pulling in all of these
different influences to automatically
generate kids worst nightmares and this
stuff really really does affect small
children right parents report their
children being traumatized becoming
afraid of the dark becoming afraid of
their favorite cartoon characters if you
take one thing away from this it's that
if you have small children keep them the
hell away from YouTube right
but the other thing the thing that
really gets gets to me about this is
this is that I'm not sure we even really
understand how we got to this point
right and we've taken all of this
influence all of these things and
managed them together in a way that no
one really intended and yet this is also
the way that we're building the entire
world we're taking all of this data a
lot of it bad data a lot of historical
data full of kind of prejudice full of
full of all of our kind of worst
impulses of history and we're building
that into huge datasets and then we're
automating it and we're managing it
together into things like credit reports
into insurance premiums into things like
predictive policing systems right into
sentencing guidelines this is the way
we're actually constructing the world
today out of this data and I don't know
what's worse that we built a system that
seems to be entirely optimized for the
absolute worst aspects of human behavior
or that we seem to have done it by
accident right without even realizing
that we were doing it because we didn't
really understand the systems that we
were building and we didn't really
understand how to do anything
differently with it there's a couple of
things I think that really seem to be
driving this most fully out on YouTube
and the first of those is advertising
right which is the kind of monetization
of attention without any real other
variables at work any care for the
people who are actually developing this
content
all right the centralization of the
power the separation of those things and
I think however you feel about the use
of advertising to kind of support stuff
the sites of grown men in diapers
rolling around in the sand in the hope
that an algorithm that they don't really
understand will give them money for it
suggests that this probably isn't the
thing that we should be basing our
society and culture upon and the way in
which we should be funding it and the
other thing that's the kind of major
driver of this is automation which is
kind of the deployment of all this
technology as soon as it arrives without
any kind of oversight and then once it's
out there kind of throwing up our hands
and going hey it's not us it's the
technology like we know we're not
involved in it that's not really good
enough because this stuff isn't just
algorithmically governed it's also
algorithmically policed all right when
YouTube first started to pay attention
to this the first thing they said they
do
it was that they deploy better machine
learning algorithms to to moderate the
content
well machine learning if you as any
expert in it will tell you is basically
what we've started to call software that
we don't really understand how it works
right and and I think we have enough of
that already
right we shouldn't be leaving this stuff
up to AI to decide what's appropriate or
not because we know what happens it'll
start censoring other things it'll start
censoring queer content it all starts
censoring kind of legitimate public
speech what's allowed in these kind of
discourse is it shouldn't be something
that's left up to unaccountable systems
it's part of a discussion all of us
should be having but I'd leave a
reminder that the alternative isn't very
pleasant either right YouTube also
announced recently that they're going to
release a version of their kids app that
would be entirely moderated by humans
and Facebook Zuckerberg said much the
same thing at the Congress were
impressed about how they were going to
moderate their stuff he said they'd have
humans doing it and what that really
means is that instead of having like
toddlers being the first person to see
this stuff you're gonna have underpaid
precarious contract workers without
proper mental health support being
damaged by it as well right and I think
we can all do quite a lot better than
that the thought I think that brings
those those two things together really
for me I think that the rings that were
together as agency it's like how much we
really understand by agency I mean how
we know how to act in our own best
interests right which it's almost
impossible to do in these systems that
we don't really fully understand
inequality of power always leads to
violence and we can see inside these
systems that inequality of understanding
does the same thing there's one thing we
can do to start to improve these systems
is to make them more legible to the
people who use them so that all of us
have a common understanding of what's
actually going on here right the thing
though I think most about these systems
is that well this isn't as I hope I've
explained really about YouTube it's
about everything these issues of
accountability and agency of opacity and
complexity at the violence and
exploitation that inherently results
from the kind of concentration of power
in a few hands these are much much
larger issues
right and they're their issues not just
of YouTube and not just of technology in
general and they're not even new they've
been with us for ages but we finally
built this system this global system the
Internet right that's actually showing
them to us in this extraordinary way
making them undeniable technology has
this extraordinary capacity to both kind
of instantiate and continue all of our
kind of most extraordinary and often
hidden kind of desires and biases and
encoding them into the world but it also
writes them down so that we can see them
right so that we can't pretend they
don't exist anymore right we need to
stop thinking about technology as a
solution to all of our problems but
think of it as a guide to what those
problems actually are so we can start
thinking about them possibly and start
to address them thank you very much
[Music]
James thank you for coming and giving us
that talk so it's interesting like when
you think about the films where the
robotic overlords take over it's all a
bit more glamorous than what you're
describing but I wonder you know in
those films you have like the resistance
mounting like where is there a
resistance mounting towards this stuff
do you see any kind of positive signs
green shoots of resistance I don't know
about kind of direct resistance because
I think this stuff is super long term I
think it's baked into culture in really
kind of deep ways and a friend of mine
Eleanor Souter always says that any
technological problems of sufficient
scale and scope are political problems
first of all right so all of these
things we're working to address within
this are not going to be addressed just
by kind of building the technology
better but actually by changing the
society that's kind of producing these
technologies so know right now I think
we've got a hell of a long way to go but
as I said I think by kind of unpacking
them by explaining them by talking about
them super honestly we can actually
start to at least begin that process and
so when you talk about legibility and
digital literacy like I find it
difficult to imagine that we need to
place the burden of digital literacy on
users themselves but whose
responsibility is education in this in
this new world I again I think this
responsibility is kind of to all of us
right that everything we do everything
we build everything we make needs to be
made in a kind of consensual discussion
with everyone who's avoiding it right
that we're not building systems that are
intended to like trick or kind of
surprise people into doing the right
thing or it's a kind of you know but
there actually is involved in every
stepping kind of educates them because
each of these systems is educational
that's what I'm kind of hopeful about
about even this really grim stuff is
that if you can take it and look at it
properly it's actually in itself as a
piece of Education that allows you to
start seeing how complex systems come
together and work and maybe be able to
apply that knowledge elsewhere in the
world James it's such an important
discussion and I know many people here
are really open and prepared to have it
so thanks for starting off our morning
thanks very much for anything
[Applause]
[Music]