Automation is Magic
The Messy Business of Security Economics.
There is no such thing as security.
I’m not being a realist here (“there are no sure bets”) nor is this mere nihilism (“you will never be safe!”).
There is no such thing as security in the abstract.You cannot be generically secure — you can only be secure from something. A sprinkler system increases your security from fires, but not burglars. Not only that, but a sprinkler system reduces your security from water-damage.
Now, a burglar alarm makes you more secure from burglars — but it makes burglars less secure from the criminal justice system. Security isn’t just contextual, it’s sometimes zero-sum, where security improvements for some are security reductions for others.
Few of us have sympathy for the plight of the poor burglar whose security is being whittled away by Big Alarm, but consider another security measure: bossware, automation systems that allow your boss to count your keystrokes, track your eye-movements, listen to your surroundings and read all your communications. Bossware increases your boss’s security by finking you out every time you take a moment to gather your equilibrium, say, by looking up for a moment, or alt-tabbing to social media for a change of pace. Bossware decreases your security by putting you in jeopardy of losing your job or having you pay docked every time you take a breather to protect your mental health.
You just can’t design a new security measure without thinking about what risk you want to mitigate. The better that understanding, the better the mitigation.
That’s where “security economics” enters the picture. As Ross Anderson says in his indispensable Economics and Security Resource Page:
Do we spend enough on keeping ‘hackers’ out of our computer systems? Do we not spend enough? Or do we spend too much? For that matter, do we spend too little on the police and the army, or too much? And do we spend our security budgets on the right things?
In its simplest form, security economics tries to balance an attacker’s gains with their costs. If breaking into a safe would cost $1,000,000 in bribes, tools, and logistics, then so long as the safe’s contents are worth $999,999.99 or less, you don’t have to worry about (rational) safebreakers.
This is an exciting idea, critical to fleshing out the “what do I want to be secure from?” question. But as simple as it sounds to ask “What’s are the contents of my safe worth to an attacker?” and “What will it cost an attacker to break into my safe?” both questions are highly contextual.
Take the question of the worth of some attack to an some attacker. A burglar who wants to steal the emerald necklace in your safe in order to fence it values that necklace at whatever the fence will pay for it.
But what about your embittered distaff cousin who is furious that your grandmother left the emerald necklace to you, and not them? For them, the necklace might be a prize beyond value, because the necklace comes with the knowledge that you, their arch-nemesis, will never feel fully secure in your home again.
More concretely, if you’re designing a cryptocurrency on the assumption that no one will spend a billion dollars on the computing power needed to take over a proof-of-work network that only secures half a billion dollars’ worth of cryptocurrency, you really need to consider that you have adversaries who might want to steal all your coins for reasons unrelated to their value.
For example, the Chinese government might decide that spending a billion dollars to wreck half a billion dollars’ worth of coins is a steal, if it scares corrupt officials out of using cryptocurrency to smuggle trillions past the country’s exchange controls.
Understanding the value of an attack is hard — but so is understanding the cost of an attack.
Maybe the safecracker who’s gunning for your emerald necklace has some blackmail dirt on your butler and can get the combination out of him for the cost of a menacing phone-call and a manila envelope full of compromising photos. For them, breaking into the safe is cheaper than buying a fancy latte at Starbucks.
Or maybe the Chinese spy agency that wants to wreck your blockchain knows about a defect in the mining rigs you’re using and they can hijack them or poison them, making it far cheaper to run a 51% attack.
Radical shifts in the cost of an attack are always an exciting and terrible time for security. When a collection of CIA cyberweapons leaked online, any cyber-criminal with half a brain could suddenly wield devastating, destructive tools. The ransomware epidemic that followed saw petty criminals hijacking entire hospitals, city governments, oil pipelines and more.
The most common source of rapid shifts in the cost of carrying out an attack isn’t blackmail dirt on butlers or secret bugs in blockchain mining-rigs or CIA cyberweapons.
When attacks suddenly get cheaper, the most common culprit is automation.
One of my favorite movies is 1985’s Real Genius, a nerdy heist movie starring Val Kilmer. One of the movie’s B-plots is the mystery of “Lazlo,” an enigmatic tinkerer who lives in a secret lair beneath a dorm at a university that’s a thinly veiled version of Caltech. When we first encounter Lazlo’s lair, we see that he has built an array of pen-plotters that are scribbling across successive waves of postcards, scrawling on them and then ejecting them into a hopper.
Later, Lazlo explains what he’s doing with all those postcards:
Lazlo: I thought you might need some help with the test, so I dug into the computer and got every question Hathaway ever asked on every final he’s ever given.
Chris: Gee, I, I didn’t get you anything. Are those they?
Lazlo: No. These are entries into the Frito-Lay Sweepstakes. “No purchase necessary, enter as often as you want” — so I am.
Chris: That’s great! How many times?
Lazlo: Well, this batch makes it one million six hundred and fifty thousand. I should win thirty-two point six percent of the prizes, including the car.
Chris: That kind of takes the fun out of it, doesn’t it?
Lazlo: They set up the rules, and lately I’ve come to realize that I have certain materialistic needs.
Lazlo’s built the pen-plotter array so that he can “hand-write” all of his postcard entries. In the movie’s denouement, he drives off with all the prizes he’s calculated he’ll win, and more.
Frito Lay understood that an “enter as often as you want” sweepstakes could be attacked by someone who mass-entered the competition, so they instituted a security measure: the requirement that entries be handwritten. Lazlo invented a handwriting machine, and cleaned them out.
The Penguin : What about the documents that prove you own half the firetraps in Gotham City?
Maximillian ‘Max’ Shreck : If there were such documents — and that’s not an admission — I would have seen to it they were shredded.
The Penguin : Ah, good idea!
[he reaches into the stocking and pulls out a sheaf of documents]
The Penguin : A lot of tape and a little patience make all the difference.
There’s a pivotal scene in Warner Brothers’ 1992 Batman Returns, where Danny DeVito, playing the Penguin, confronts Christopher Walken, playing Max Schreck, and asks him about “the documents that prove you own half the firetraps in Gotham City.” Walken replies that such documents (if they existed) would have been shredded. DeVito triumphantly produces a sheaf of laboriously taped-together, formerly shredded documents and brandishes them, saying, “ A lot of tape and a little patience make all the difference.”
The security of document shredding assumes that the cost of reassembling shredded strips of paper is greater than the value of those papers. For The Penguin, who is motivated by a mix of personal, idiosyncratic factors (his spite towards Mac Schreck) and a big cash payoff (control over Gotham’s criminal networks), a few hundred hours’ worth of piecing together Schreck’s shredded files is well worth it.
Today, most people who bother to shred their documents, use cross-cut shredders that produce something that looks a lot more like confetti than the fettuccine ribbons the Penguin taped back together.
As a countermeasure, the switch to cross-cut might have been premature, as automation has not yet turned DeVito’s “lot of tape and a little patience” into an app. In 2008, the German government announced that they were automating the manual reassembly the millions of documents the Stasi, East Germany’s hated secret police, which were shredded in haste as the Berlin Wall fell. In 2018, they announced that the project had failed, with a decade’s worth of work yielding very little in the way of automation tools for reassembling shredded paperwork.
For many of us, our first introduction to automation attacks came in 1983's Superman III, where Richard Pryor conceives of a scheme to siphon off all the fractional pennies that a bank’s computer rounds down in its calculations and stash them in a secret account.
This attack — formally called “salami slicing” — became a pop-culture staple (it was the McGuffin of 1999’s Office Space), and it continues to pay dividends. There’s a form of high-frequency trading that bears a striking resemblance to it:
I don’t think that the timing of these movies about automation attacks on security economics is an accident. As computers got faster and cheaper, we saw a great democratization of cybercrime.
Digital security systems that were designed to withstand decades’ worth of computational brute-force attacks were shredded like wet toilet paper as computers got much faster.
Take the humble salted password file, once the gold standard in password handling (replacing the unsalted password file, wherein computers stored their users’ passwords in a human-readable format).
Advances in computing power, particularly graphics cards, made it possible to produce “rainbow tables” — massive files containing every human-memorizable password, in hashed and unhashed form (“a lot of tape and a little patience,” indeed).
Suddenly, anyone who hadn’t upgraded to “salted hashes” (once considered wild overkill) was conducting themselves with a recklessness that can only be called negligence.
But remember, there is no such thing as security. Something that makes your enemy more secure makes you less secure.
Take DoNotPay (“the world’s first robot lawyer”), which started off as a tool to automate contesting parking tickets. Founded by then-student Joshua Browder, DoNotPay expanded into automating a variety of bureaucratic tasks whose cumbersome nature was a deliberate strategy to discourage people from getting their due.
In 2016, DoNotPay automated the brutal, Kafka-esque process of applying for benefits in England, so homeless and precarious people could cut through the red-tape thicket that the Conservative government had thrown in their path.
Then, in 2017, Equifax dumped the personal information of nearly every person in America — as well as tens of millions of people abroad — and faced a slap on the wrist from regulators. Individuals were entitled to damages for the breach, but collecting involved suing. DoNotPay automated the process of filing a small-claims court suit against Equifax.
US airlines are required to issue you a refund if the price of your ticket drops between the time you purchase it and the day you fly. This process is incredibly, deliberately cumbersome — unless you use DoNotPay, which automates it.
Many companies offer guarantees and refunds if you call in and speak to a customer service rep — then they staff their customer service phone-banks with skeleton crews, ensuring that you must wait for hours to claim small-dollar promises. DoNotPay waits on hold for you, upending the security economics of long hold times.
DoNotPay is basically dozens of automation attacks offered for $3/month. You know all those pesky services that make signing up as easy as a greased slide, but make resigning a ten-mile crawl over broken glass? DoNotPay automates quitting them.
Of course, the easiest way to resign from a service is often to cancel the credit-card you used at sign-up, but then you have to go through the hassle of changing the credit-card everywhere else. DoNotPay offers burner, one-time-use virtual credit cards, so you can sign up for an auto-renewing 30-day trial with a credit-card that expires after 29 days.
An automation attack doesn’t need to be fully automated to be a game-changer.
The scourge of modern legal rights is the “binding arbitration waiver,” a clause in a click-through agreement that says you surrender your right to sue the company you’re doing business with, no matter how badly they mistreat you. Instead, you agree to go to “arbitration,” in which a fake corporate judge, paid by the company that wronged you, decides whether they’re in the wrong.
This doesn’t just show up in the license agreements for products and services — it’s also a fixture in work contracts with giant global “gig work” platforms, who use it to ensure that their workers can’t sue them for abusing them, stealing their wages, or subjecting them to unsafe conditions.
Enter “mass arbitration,” a tactic developed by a small group of highly specialized law firms that substantially automates filing arbitration claims on behalf of large groups of people who’ve been wronged by a company. It’s still more expensive than filing a class-action suit, but — crucially — it’s also much more expensive to defend against.
In 2018, Uber was hit by the first wave of mass-arbitration wave, as thousands of California Uber drivers exercised the clause in the contract Uber had rammed down their throats, demanding that Uber hire thousands of arbitrators to hear each of their cases individually. Then Uber was hit again by restaurateurs that Uber nonconsensually opted into its pandemic delivery services. The company settled the drivers’ claims for $146,000,000. It’s still thrashing over the restaurant claims, where it owes $100,000,000 in arbitration fees alone , not counting any damages.
The law-firm that hit Uber with 31,000 arbitration claims is called Consovoy McCarthy. They have 13 employees (and a whole lot of automation).
Then there’s Intuit, the company that convinced the IRS not to offer Americans free, pre-completed tax-returns, insisting that its Turbotax “Free File” program would help people who needed it. Instead, Turbotax used scammy, deceptive tactics to trick people into spending hours entering their tax data into its forms and then telling them they didn’t qualify and had to pay to finish their filing.
Intuit thought they could get away with it because their website made you click through an arbitration waiver in order to start your tax-return. Now the company is facing more than 100,000 arbitration claims, thanks to the law firm Keller Lenkner (which has successfully used mass arbitration to force Postmates and Doordash to make good on the wages they stole from their workers).
Intuit is so desperate they tried to get a judge to declare that their binding arbitration waiver wasn’t binding, in the hope that they’d face a good, old fashioned class action suit from the customers they ripped off. The judge told ’em, “Nope, you told your customers they were subject to binding arbitration, and that cuts both ways” (actually, it was even better than this! US District Court Judge Charles Breyer told Intuit’s lawyers, “Intuit was, in Hamlet’s words, hoisted by their own petard…arbitration is the petard that Intuit now faces”).
Congress has dragged its feet on forced arbitration for a decade. At this rate, automation might actually kill forced arbitration before Congress does. In 2021, Amazon amended the Alexa terms of service to remove forced arbitration, in order to avert the thread of automated mass arbitration.
Cory Doctorow (craphound.com) is a science fiction author, activist, and blogger. He has a podcast, a newsletter, a Twitter feed, a Mastodon feed, and a Tumblr feed. He was born in Canada, became a British citizen and now lives in Burbank, California. His latest nonfiction book is How to Destroy Surveillance Capitalism. His latest novel for adults is Attack Surface. His latest short story collection is Radicalized. His latest picture book is Poesy the Monster Slayer. His latest YA novel is Pirate Cinema. His latest graphic novel is In Real Life. His forthcoming books include Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid (with Rebecca Giblin), a book about artistic labor market and excessive buyer power; Red Team Blues, a noir thriller about cryptocurrency, corruption and money-laundering (Tor, 2023); and The Lost Cause, a utopian post-GND novel about truth and reconciliation with white nationalist militias (Tor, 2023).
