This is the text version of remarks given by the author on June 26, 2016, at a panel on the Moral Economy of Tech at the SASE conference in Berkeley to an audience of social scientists. The other panel participants were Kieran Healy, Stuart Russell and AnnaLee Saxenian. This piece originally appeared on the author’s blog, Idle Words.
I am only a small minnow in the technology ocean, but since it is my natural habitat, I want to make an effort to describe it to you.
As computer programmers, our formative intellectual experience is working with deterministic systems that have been designed by other human beings. These can be very complex, but the complexity is not the kind we find in the natural world. It is ultimately always tractable. Find the right abstractions, and the puzzle box opens before you.
The feeling of competence, control, and delight in discovering a clever twist that solves a difficult problem is what makes being a computer programmer sometimes enjoyable.
“The real world is a stubborn place. It is complex in ways that resist abstraction and modeling.”
But as anyone who’s worked with tech people knows, this intellectual background can also lead to arrogance. People who excel at software design become convinced that they have a unique ability to understand any kind of system at all, from first principles, without prior training, thanks to their superior powers of analysis. Success in the artificially constructed world of software design promotes a dangerous confidence.
Today, we are embarked on a great project to make computers a part of everyday life. As Marc Andreessen memorably frames it, “Software is eating the world.” And those of us writing the software expect to be greeted as liberators. Our intentions are simple and clear. First we will instrument, then we will analyze, then we will optimize. And you will thank us.
But the real world is a stubborn place. It is complex in ways that resist abstraction and modeling. It notices and reacts to our attempts to affect it. Nor can we hope to examine it objectively from the outside, any more than we can step out of our own skin.
The connected world we’re building may resemble a computer system, but really it’s just the regular old world from before, with a bunch of microphones and keyboards and flat screens sticking out of it. And it has the same old problems.
Approaching the world as a software problem is a category error that has led us into some terrible habits of mind.
Bad mental habits
“Treating the world as a software project gives us a rationale for being selfish.”
First, programmers are trained to seek maximal and global solutions. Why solve a specific problem in one place when you can fix the general problem for everybody, and for all time? We don’t think of this as hubris, but as a laudable economy of effort. And the startup funding culture of big risk, big reward encourages this grandiose mode of thinking. There is powerful social pressure to avoid incremental change, particularly any change that would require working with people outside tech and treating them as intellectual equals.
Second, treating the world as a software project gives us a rationale for being selfish. The old adage has it that if you are given 10 minutes to cut down a tree, you should spend the first five sharpening your ax. We are used to the idea of bootstrapping ourselves into a position of maximum leverage before tackling a problem.
In the real world, this has led to a pathology where the tech sector maximizes its own comfort. You don’t have to go far to see this. Hop on BART after the conference and take a look at Oakland, or take a stroll through downtown San Francisco and try to persuade yourself you’re in the heart of a boom that has lasted for 40 years. You’ll see a residential theme park for tech workers, surrounded by areas of poverty and misery that have seen no benefit and ample harm from our presence. We pretend that by maximizing our convenience and productivity, we’re hastening the day when we finally make life better for all those other people.
Third, treating the world as software promotes fantasies of control. And the best kind of control is control without responsibility. Our unique position as authors of software used by millions gives us power, but we don’t accept that this should make us accountable. We’re programmers — who else is going to write the software that runs the world? To put it plainly, we are surprised that people seem to get mad at us for trying to help.
“People obsessed with control have to eventually confront the fact of their own extinction.”
Fortunately, we are smart people and have found a way out of this predicament. Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It’s a clean, mathematical apparatus that gives the status quo the aura of logical inevitability. The numbers don’t lie.
Of course, people obsessed with control have to eventually confront the fact of their own extinction. The response of the tech world to death has been enthusiastic. We are going to fix it. Google Ventures, for example, is seriously funding research into immortality. Their head VC will call you a ‘deathist’ for pointing out that this is delusional.
Such fantasies of control come with a dark side. Witness the current anxieties about an artificial super intelligence, or Elon Musk’s apparently sincere belief that we’re living in a simulation. For a computer programmer, that’s the ultimate loss of control. Instead of writing the software, you are the software.
We obsess over these fake problems while creating some real ones.
In our attempt to feed the world to software, techies have built the greatest surveillance apparatus the world has ever seen. Unlike earlier efforts, this one is fully mechanized and in a large sense autonomous. Its power is latent, lying in the vast amounts of permanently stored personal data about entire populations.
We started out collecting this information by accident, as part of our project to automate everything, but soon realized that it had economic value. We could use it to make the process self-funding. And so mechanized surveillance has become the economic basis of the modern tech industry.
Surveillance capitalism
Surveillance capitalism has some of the features of a zero-sum game. The actual value of the data collected is not clear, but it is definitely an advantage to collect more than your rivals do. Because human beings develop an immune response to new forms of tracking and manipulation, the only way to stay successful is to keep finding novel ways to peer into people’s private lives. And because much of the surveillance economy is funded by speculators, there is an incentive to try flashy things that will capture the speculators’ imagination, and attract their money.
This creates a ratcheting effect where the behavior of ever more people is tracked ever more closely, and the collected information retained, in the hopes that further dollars can be squeezed out of it.
Just like industrialized manufacturing changed the relationship between labor and capital, surveillance capitalism is changing the relationship between private citizens and the entities doing the tracking. Our old ideas about individual privacy and consent no longer hold in a world where personal data is harvested on an industrial scale.
Those who benefit from the death of privacy attempt to frame our subjugation in terms of freedom, just like early factory owners talked about the sanctity of contract law. They insisted that a worker should have the right to agree to anything, from 16-hour days to unsafe working conditions, as if factory owners and workers were on an equal footing.
Companies that perform surveillance are attempting the same mental trick. They assert that we freely share our data in return for valuable services. But opting out of surveillance capitalism is like opting out of electricity, or cooked foods — you are free to do it in theory. In practice, it will upend your life.
Many of you had to obtain a US visa to attend this conference. The customs service announced yesterday it wants to start asking people for their social media profiles. Imagine trying to attend your next conference without a LinkedIn profile, and explaining to the American authorities why you are so suspiciously off the grid.
The reality is, opting out of surveillance capitalism means opting out of much of modern life.
We’re used to talking about the private and public sector in the real economy, but in the surveillance economy this boundary doesn’t exist. Much of the day-to-day work of surveillance is done by telecommunications firms, which have a close relationship with government. The techniques and software of surveillance are freely shared between practitioners on both sides. All of the major players in the surveillance economy cooperate with their own country’s intelligence agencies, and are spied on (very effectively) by all the others.
As a technologist, this state of affairs gives me the feeling of living in a forest that is filling up with dry, dead wood. The very personal, very potent information we’re gathering about people never goes away, only accumulates. I don’t want to see the fire come, but at the same time, I can’t figure out a way to persuade other people of the great danger.
So I try to spin scenarios.
The inevitable list of scary scenarios
One of the candidates running for President this year has promised to deport 11 million undocumented immigrants living in the United States, as well as block Muslims from entering the country altogether. Try to imagine this policy enacted using the tools of modern technology. The FBI would subpoena Facebook for information on every user born abroad. Email and phone conversations would be monitored to check for the use of Arabic or Spanish, and sentiment analysis applied to see if the participants sounded ‘nervous.’ Social networks, phone metadata, and cell phone tracking would lead police to nests of hiding immigrants.
We could do a really good job deporting people if we put our minds to it.
Or consider the other candidate running for President, the one we consider the sane alternative, who has been a longtime promoter of a system of extrajudicial murder that uses blanket surveillance of cell phone traffic, email, and social media to create lists of people to be tracked and killed with autonomous aircraft. The system presumably includes points of human control (we don’t know because it’s secret), but there’s no reason in principle it could not be automated. Get into the wrong person’s car in Yemen, and you lose your life.
That this tool chain for eliminating enemies of the state is only allowed to operate in poor, remote places is a comfort to those of us who live elsewhere, but you can imagine scenarios where a mass panic would broaden its scope.
Or imagine what the British surveillance state, already the worst in Europe, is going to look like in two years, when it’s no longer bound by the protections of European law, and economic crisis has driven the country further into xenophobia.
Or take an example from my home country, Poland. Abortion has been illegal in Poland for some time, but the governing party wants to tighten restrictions on abortion by investigating every miscarriage as a potential crime. Women will basically be murder suspects if they lose their baby. Imagine government agents combing your Twitter account, fitness tracker logs, credit card receipts, and private communications for signs of potential pregnancy, with the results reported to the police to proactively protect your unborn baby.
We tend to imagine dystopian scenarios as one where a repressive government uses technology against its people. But what scares me in these scenarios is that each one would have broad social support, possibly majority support. Democratic societies sometimes adopt terrible policies.
When we talk about the moral economy of tech, we must confront the fact that we have created a powerful tool of social control. Those who run the surveillance apparatus understand its capabilities in a way the average citizen does not. My greatest fear is seeing the full might of the surveillance apparatus unleashed against a despised minority in a democratic country.
What we’ve done as technologists is leave a loaded gun lying around, in the hopes that no one will ever pick it up and use it.
Conclusion
The first step toward a better tech economy is humility and recognition of limits. It’s time to hold technology politically accountable for its promises. I am very suspicious of attempts to change the world that can’t first work on a local scale. If after decades we can’t improve quality of life in places where the tech élite actually lives, why would we possibly make life better anywhere else? We should not listen to people who promise to make Mars safe for human habitation, until we have seen them make Oakland safe for human habitation. We should be skeptical of promises to revolutionize transportation from people who can’t fix BART, or have never taken BART. And if Google offers to make us immortal, we should check first to make sure we’ll have someplace to live.
Techies will complain that trivial problems of life in the Bay Area are hard because they involve politics. But they should involve politics. Politics is the thing we do to keep ourselves from murdering each other. In a world where everyone uses computers and software, we need to exercise democratic control over that software.
Second, the surveillance economy is way too dangerous. Even if you trust everyone spying on you right now, the data they’re collecting will eventually be stolen or bought by people who scare you. We have no ability to secure large data collections over time.
“Let’s at least learn from our past, so we can fail in interesting new ways.”
The goal should be not to make the apparatus of surveillance politically accountable (though that is a great goal), but to dismantle it. Just like we don’t let countries build reactors that produce plutonium, no matter how sincere their promises not to misuse it, we should not allow people to create and indefinitely store databases of personal information. The risks are too high. I think a workable compromise will be to allow all kinds of surveillance, but limit what anyone is allowed to store or sell.
More broadly, we have to stop treating computer technology as something unprecedented in human history. Not every year is Year Zero. This is not the first time an enthusiastic group of nerds has decided to treat the rest of the world as a science experiment. Earlier attempts to create a rationalist Utopia failed for interesting reasons, and since we bought those lessons at a great price, it would be a shame not to learn them.
There is also prior art in attempts at achieving immortality, limitless wealth, and galactic domination. We even know what happens if you try to keep dossiers on an entire country.
If we’re going to try all these things again, let’s at least learn from our past, so we can fail in interesting new ways, instead of failing in the same exasperating ways as last time.
