see also: Shoshanna Zuboff (who wrote the seminal work on surveillance capitalism), Don Norman, Dystopia vs. Utopia Book List: A Fight to the Finish, surveillance capitalism dictionary
Some takeaways:
- surveillance won’t be obvious and overt like in Orwell’s classic totalitarian novel 1984 — it’ll be covert and subtle (“more like a spider’s web”)
- social networks use persuasion architecture — the same cloying design aesthetic that puts gum at the eye level of children in the grocery aisle
Example:
AI modeling of potential Las Vegas ticket buyers
The machine learning algorithms can classify people into two buckets, “likely to buy tickets to Vegas” and “unlikely to” based on exposure to lots and lots of data patterns. Problem being, it’s a black box and no one — not even the computer scientists — know how it works or what it’s doing exactly.
So the AI may have discovered that bipolar individuals just about to go into mania are more susceptible to buying tickets to Vegas — and that is the segment of the population they are targeting: a vulnerable set of people prone to overspending and gambling addictions. The ethical implications of unleashing this on the world — and routinely using and optimizing it relentlessly — are staggering.
Profiting from extremism
“You’re never hardcore enough for YouTube” — YouTube gives you content recommendations that are increasingly polarized and polarizing, because it turns out that preying on your reptilian brain makes you keep clicking around in the YouTube hamster wheel.
The amorality of AI — “algorithms don’t care if they’re selling shoes, or politics.” Our social, political, and cultural flows are being organized by these persuasion architectures — organized for profit; not for the collective good, not for public interests, not subject to our political will anymore. These powerful surveillance capitalism tools are running mostly unchecked, with little oversight and with few people minding the ethics of the stores of essentially a cadre of Silicon Valley billionaires.
Intent doesn’t matter — good intentions aren’t enough; it’s the structure and business models that matter. Facebook isn’t a half trillion dollar con: its value is in its highly effective persuasion power, which is highly troubling and concerning in a supposedly democratic society. Mark Zuckerberg may even ultimately mean well (…debatable), but it doesn’t excuse the railroading over numerous obviously negative externalities resulting from the unchecked power of Facebook in not only the U.S., but in countries around the world including highly volatile domains.
Extremism benefits demagogues — Oppressive regimes both come to power by and benefit from political extremism; from whipping up citizens into a frenzy, often against each other as much as against perceived external or internal enemies. Our data and attention are now for sale to the highest bidding authoritarians and demagogues around the world — enabling them to use AI against us in election after election and PR campaign after PR campaign. We gave foreign dictators even greater powers to influence and persuade us in ways that benefit them at the expense of our own self-interest.