By Austin VornhagenThe lock held.
Picture a bakery that books most of its catering through ads on one big platform. One morning the ad account is gone. Disabled. The notice names a policy, not a reason. The owner hadn’t checked that account in a while, so by the time they notice, the window to ask for a review has closed.
They open a new account. It gets banned too. The platform recognized them.
Here’s the strange part. Nobody hacked anything. Nothing leaked. Every byte of that bakery’s data is sitting exactly where it’s supposed to be: encrypted, backed up, access-controlled, and perfectly secure. The system that hurt them wasn’t broken. It was working.
The bakery is a composite. Variations of its story are common enough that business owners trade recovery tips for it. And it exposes a mix-up most of us carry around without noticing.
Is it protected from strangers who want to steal, change, or destroy it?
Usually: yes.Is it protected from the people who are allowed to hold it?
That’s a different question.A lock protects the file from strangers. Nothing in the lock protects you from the file.
We’ve spent decades getting better at the first question. Most of the damage in the history of personal data comes from the second. So let’s walk the records hall from the beginning: who started keeping files on people, who those files ended up hurting, and why the fix was never just a better lock. Then we’ll design a world where a machine’s mistake about you is survivable.
The first file on you was a rumor
Before writing, information lived in relationships. In a small community, everyone knew who owed whom, who could be trusted, and who had broken a rule. Reputation made cooperation possible: you lend to the neighbor with a good name.
It also gave gossip and secrets their teeth. Nobody needed a database to threaten to reveal something, or to turn a village against someone. Information was a weapon long before it was a record.
Writing changed one thing: a fact could outlive the person who knew it. Mesopotamian scribes recorded loans, sales, ownership, and legal disputes on clay tablets, and seals identified who had authorized them. Some tablets were sealed inside clay envelopes, so tampering with the text meant breaking the outside first.
That envelope is the ancestor of every security feature you use. And notice what it protected: the authority of the record. Was it authentic? Had anyone altered it? Nobody was asking whether the person described in the tablet would be safe from the person holding it.
Keep that envelope in mind and walk forward through the hall. At each stop, ask one question: did the harm come from someone breaking in, or from whoever held the file?
Nine moments in the history of personal records, sorted by whether the harm came from a break-in or from whoever held the file
A tally of the nine episodes in this essay, not a statistic about all data harm.
The lock is invented
Scribes press contracts and court testimony into clay, then seal some tablets inside clay envelopes stamped with the parties’ seals. To change the text, you’d have to break the envelope. The lock protects the record’s authority. It says nothing about the person in it.
Protects the recordThe king takes inventory
William the Conqueror’s Domesday survey records who holds what land and what it’s worth, so the crown can see its kingdom and its tax potential. Nobody broke in. The holder of the record is the one who acts on it.
Harm came from the holderYour reputation goes on sale
Credit-reporting firms like R. G. Dun & Company sell merchants reports on strangers’ wealth, habits, and character, hearsay included. The person described has no legal right to read their own report.
Harm came from the holderRoutine records change hands
Authorities identify Jews using community membership lists, church and tax records, police registries, forced registration, and neighbors. Paper was enough. No computer, and no breach.
Harm came from the holderSurveillance as a lever
The FBI wiretaps and works to discredit Martin Luther King Jr. Its COINTELPRO operations target civil-rights and antiwar groups. The Senate’s Church Committee later documents the abuses.
Harm came from the holderThe file goes to market
The FTC finds data brokers buying and reselling information about people, inferring traits like health interests and income. In 2024 it acts against a broker over precise location data that could reveal visits to clinics, places of worship, and shelters.
Harm came from the holderFinally, a break-in
Attackers breach Ashley Madison and expose account and profile information tied to about 36 million users, by the FTC’s count. This is the kind of harm a better lock prevents.
Harm came from outsideFakes that know your name
The FBI warns that criminals use generative AI to write convincing messages, fake identities, and clone voices, including impersonating a loved one in a crisis to ask for money.
Harm came from outsideThe machine decides
An automated system flags a business account, removes it, and recognizes the owner when they try again. Every byte stays encrypted, backed up, and exactly where it belongs.
Harm came from the holderSix arrows came from the desk. Two came through the wall.
That’s not a statistic. It’s the lineup in this essay, and you could pick a different one. But try it. Most of the harms people remember from the history of records needed no break-in at all. A locked archive offers little protection from the ruler who owns the archive.
Which raises an awkward question about the word we use for all this. When a company tells you your data is “safe,” it is answering the question about the wall. The desk is a different conversation.
A stranger wrote your file. You weren’t allowed to see it.
Jump to the nineteenth century. Trade is exploding across a big young country, and a merchant in New York needs to decide whether to extend credit to a shopkeeper in Ohio he will never meet.
Enter the credit reporters. Firms like R. G. Dun & Company built networks of local correspondents who wrote up the people around them: their wealth, their business history, and their character. The surviving ledgers are full of opinion and hearsay. The people being judged had no legal right to read what was written about them.
Sit with how modern that is. Someone you’ve never met writes something about you that changes your opportunities, and you have no way to see it or correct it. No thief required. An inaccurate or prejudiced report could do its damage traveling through the intended business process, exactly as designed.
It took until 1970 for the U.S. to answer it. The Fair Credit Reporting Act gave people the right to learn what a consumer reporting agency had on file about them and to dispute what was wrong. Hold onto that: the fix for a secret file wasn’t a stronger lock on the file. It was a right for the person in it.
- Name
- Address
- Religious community
- Occupation
A routine entry, for taxes, services, or a congregation’s membership roll.
The same card becomes a way to find someone.
The twentieth century delivered the darkest version of that lesson. When Nazi authorities and their collaborators set out to identify Jews, they drew on community membership lists, church and tax records, police registries, forced registration, and the knowledge of neighbors. The U.S. Holocaust Memorial Museum emphasizes that this depended on paper and local knowledge. It needed no computers.
Information collected under one set of rules can become dangerous when the rules change. An address or a religious affiliation looks harmless on the day it’s written down. Under a persecuting regime, it’s a map. And no amount of protection against outsiders would have helped, because the people using the records were the ones in charge of them.
Mid-century intelligence agencies refined a quieter version: learn something damaging, then use exposure, or the threat of it, as a lever. It’s the logic of blackmail, run by a government. The FBI’s campaign against Martin Luther King Jr. and its COINTELPRO operations are documented examples.
Then computers arrived, and a clerk who once needed hours with a filing cabinet could search, copy, and cross-reference in moments. In 1973, a U.S. government report called Records, Computers, and the Rights of Citizens put the problem in one sentence: it was “becoming much easier for record-keeping systems to affect people than for people to affect record-keeping systems.” It proposed principles for automated records: no secret systems, a way for people to see and correct their records, and limits on using information for a new purpose. Those ideas shaped the Privacy Act of 1974, which covers certain federal agency records.
By then, two different disciplines were growing side by side, and it’s worth seeing them next to each other, because people mix them up constantly:
Ciphers, seals, locks, passwords, encryption. The U.S. adopted the Data Encryption Standard in 1977; public-key cryptography emerged publicly in the same decade and made secure online commerce possible.
Guards the wallNotice, access, correction, purpose limits, and the right to dispute a decision. The 1970 credit law and the 1973 report live here.
Guards the deskA system can have excellent security while its owner uses the information abusively.
Then the file started writing itself
The internet turned ordinary life into a stream of records. Searching, shopping, posting, and messaging all left traces, and companies could join them to offline purchases and public records.
In 2014, the Federal Trade Commission looked at the data broker industry and found companies collecting information from many sources, passing it through chains of intermediaries, and inferring things about people: health interests, income, religion. Most consumers had no idea these companies existed.
That’s a genuinely new move in the history of the file. Nobody needed you to disclose a sensitive fact anymore. They could guess it. Repeated visits to one address might suggest something about your health or your faith. The guess might be right. It might be completely wrong. Either way, someone can act on it.
Phones made location the most revealing record of all. In 2024 the FTC acted against the data broker X-Mode Social and its successor Outlogic over the sale of precise location data that could reveal visits to medical facilities, places of worship, and domestic-abuse shelters.
The wall was busy, too. The 2015 Ashley Madison breach exposed account and profile information tied to about 36 million users, according to the FTC. And digital copying changed what a leak means: recover a stolen folder and you may have the only copy. Recover a stolen database, and you have proved nothing about who else still has it. Ransomware added a second squeeze, locking an organization out of its own data while threatening to publish it. Backups can restore operations. They can’t un-publish anything.
Now AI is lowering the price of personalized deception. In December 2024 the FBI warned that criminals use generative AI for convincing messages, fake identities, and cloned voices. Picture a scammer who knows your child’s name and uses a synthetic voice to make an emergency call sound real. The true facts and the fake voice reinforce each other. AI doesn’t need to know you perfectly to hurt you. Partial knowledge is enough.
Line up five thousand years of this and the same handful of moves keeps coming back. Tap a drawer.
Coercion
“Do what I want, or I reveal this.”
Exclusion
“This record or classification means you can’t participate.”
Impersonation
“I know enough about you to convince someone I am you.”
Manipulation
“I know which fear, need, or relationship will move you.”
Targeting
“I can identify and locate you, or the people connected to you.”
Reputational attack
“I can circulate something damaging, false, or stripped of context.”
Look at which drawer the bakery is stuck in. Exclusion. And notice something new about it. Every other harm in this history needed a person to decide to act. The bakery’s didn’t.
How a flag becomes a life sentence
Let’s open the bakery’s case file and watch what goes into it. None of the steps is outrageous on its own. That’s what makes the combination so hard to escape.
The bakery's case file, filling up one stamp at a time
A system decides you’re a risk
Some signal trips a detector: an old account connection, a login pattern, an ad that resembles something bad. Nobody you can talk to has looked at it yet.
The judgment lands on your livelihood
The ad account is how the bakery books its catering orders. When it goes dark, the phone stops ringing. The restriction isn’t a warning. It’s a closed storefront.
The reason is a category, not a fact
The notice cites a policy, not the evidence. Detection details stay confidential, partly for a real reason: scammers would study them. But you can’t disprove a label.
A window closes while you aren’t looking
Meta’s help pages give suspended accounts 180 days to appeal, and say the window varies by region. Miss it and the page says the account is permanently disabled, with no further review.
Start over, and the system knows you
Meta says it collects device identifiers, IP addresses, and network information, and may restrict business assets connected to accounts that evade its rules. A fresh account inherits the old verdict.
Here’s the precise version of the device claim, because it matters. Collecting device and network identifiers supports real concern about linking accounts together. It does not show that a platform can see everything on every device in your home, and it doesn’t tell you which signal caused a particular ban. The argument is stronger without the exaggeration.
And to be fair to the platforms: there is a legitimate reason to stop banned scammers from returning under new names. That is exactly why the correction process matters more, not less. The better a system gets at recognizing you, the more expensive its mistakes about you become.
Recognizing you accurately does not prove the original judgment was accurate.
A system can get extremely good at finding the same person again while staying bad at checking whether it was right about them the first time. To see why, climb the ladder that turns a device ID into a verdict.
- ObservedThese accounts used the same device.
A fact a log can show. It could be a shared family tablet, a reused phone, or a café’s computer.
- InferredThese accounts belong to the same person.
A guess layered on the fact. Usually reasonable, sometimes wrong.
- InferredThat person committed fraud.
A much bigger claim, about intent and conduct, resting on the first two.
- ConsequenceEvery business tied to them is excluded.
A punishment, which needs its own justification even if all three claims are true.
Each rung adds a claim the one below it doesn’t prove. Pick a mode, then overturn the fraud finding.
Two more things hide in that ladder. If five warning signals all trace back to the same original flag, they aren’t five pieces of evidence. They’re one, counted five times. And a deadline does nothing to the rungs at all.
A missed appeal deadline is not evidence that the accusation was true.
A system that treats the deadline as a reason to keep a possibly mistaken exclusion forever has an accountability problem, whether or not anyone at the company means any harm. They don’t have to. It’s enough to accept some false positives, make review hard to reach, and let the people on the wrong end absorb the losses.
Seven fixes for a world where a machine’s mistake is survivable
You can’t fix this by banning automation. Platforms really do face scammers, and people really do get hurt by them. The goal is narrower, and more achievable, than “never make mistakes.” Here is the guarantee I’d build everything around:
An automated mistake should have a limited reach, a limited duration, and a reliable route back.
People could still face restrictions for harmful conduct. But keeping a job, receiving essential services, or running a legitimate business wouldn’t depend on persuading an unreachable computer that it misunderstood you. These are designs to develop and test, not proven systems, and some pieces already exist in other forms.
Replace the light switch with a control panel
Most accounts work like a single switch: on or off. Split access into separate capabilities, and require a restriction to target only the capability creating the danger. A suspicious new campaign is a reason to pause new campaigns. It is not, by itself, a reason to delete your records and cut off your customers.
For essential services, the protected minimum gets stronger: a payment app could hold one suspicious transfer while you keep access to verified, uncontested funds.
- See your records and history
- Answer existing customers
- Run already-approved ads
- Launch new campaigns
- Change payment details
- Keep your page and reviews
- See your records and history
- Answer existing customers
- Run already-approved ads
- Launch new campaigns
- Change payment details
- Keep your page and reviews
Both accounts are running normally. Press the trigger.
Make every restriction expire unless a person renews it
Write each consequential restriction like a warrant with a date on it. The enforcement software refuses to apply one that has lapsed. The detector that imposed it can’t quietly renew its own judgment; renewal takes a new decision by someone accountable, and long-term exclusion from an important service takes an independent one.
- Justified by
- Suspected fraudulent campaign, flagged by an automated system
- Restricts
- Launching new campaigns only
- Expires
- Automatically, on a stated date
- Renewable by
- A named human reviewer, not the detector
- Ends early if
- Business verification passes, or the evidence is rebutted
NIST’s AI Risk Management Framework already asks organizations to plan for appeal, human override, and shutting off systems that misbehave. An expiry date goes a step further: it makes continued enforcement depend on someone actively choosing it. In the bakery’s case, noticing an old restriction months later would trigger a fresh look, instead of turning an unexamined accusation into permanent proof.
Independence matters. A second AI reading the first AI’s conclusion is weak protection. Real reviewers need the underlying evidence, the authority to reverse, and funding that doesn’t depend on pleasing the platform.
Let people prove safe conduct without first winning an argument about the past
This is the idea I find most promising. Sometimes nobody can settle whether an old account was hacked, fraudulent, or wrongly flagged. Today, that uncertainty means “no” forever. Change the question to: can we build a way for this person to participate where the suspected harm is fenced in?
- VerifyProve you control the business
- SuperviseEach ad and landing page approved before it runs
- LimitSmall budgets and slow changes at first
- ExpandAccess grows with observed good conduct
- GraduateA clear end date, so probation isn’t permanent
The platform could say: “We haven’t resolved the old account issue. Here are the conditions under which your business can advertise safely today.” That swaps an impossible task, proving a negative about a years-old event, for a concrete one. It needs guardrails: the conditions should match the actual suspected danger, and waiting alone shouldn’t clear a real fraudster.
Keep facts and guesses in separate columns, and let corrections flow downstream
That’s the ladder you just climbed, turned into a rule. Every consequential decision records which rungs are observed and which are inferred, where each came from, and how uncertain it is. Signals that trace back to one flag are counted once. And when a finding is overturned, every restriction built on it is reconsidered automatically, so nobody has to appeal separately to every system that copied the mistake. That record needs strict access limits of its own, or it becomes a new blacklist.
Make whoever imposes a restriction carry part of its cost
Right now, automating enforcement saves a platform money while a mistaken exclusion costs the person on the other end. Flip the math. A pooled, independently run fund, paid for by participating providers, would pay for independent review and limited support during disputes. Providers with demonstrably poor enforcement would pay more.
The details decide whether this helps. Charge providers for every justified fraud block and they’ll stop protecting people. Judge them only by appeal reversals and they’ll hide the appeal button. So random audits would have to include people who never appealed, including people who never saw the notice. A system shouldn’t look good because the people it harmed gave up.
Prove the one fact that matters, not your whole life
A service usually needs a narrow answer. Do you control this business? Is this payment method yours? It doesn’t need a growing map of your household devices. Independently issued credentials can answer narrow questions while revealing only the necessary facts; the W3C’s work on verifiable credentials includes this kind of selective disclosure. They have limits: an authentic credential can still contain a false claim. And this must never turn into one universal “trustworthiness score.” Multiple issuers, a way to correct them, and an alternative for people without a digital credential are part of the design.
Carry your own file
Every fix so far makes the person holding the file behave better. The last one asks a stranger question. What if, when you log off, your data logged off with you?
Imagine opening a social app, your information flowing in, the app personalizing everything, and then, when you close it, all of it leaving the company’s systems and coming back to a wallet in your pocket. The company never keeps the memory. You do. Scroll through one session.
One app session with a personal data wallet, step by step
The memory lives with them
Your interests, history, and the conclusions drawn from them accumulate on the company’s servers. Your phone is a window into a file you never hold.
Put the file in your pocket
A personal data wallet on your phone, with encrypted backups you control, holds your preferences and history. Apps get permission to use it. They don’t get to keep it.
Only the slice the session needs travels
Opening a video app sends the relevant preferences into a sealed workspace: either on your own device, or inside a protected server room built to forget.
The app brings the logic. You bring the memory.
The app’s recommendation software runs inside the sealed room, ranks the catalog against your preferences, and hands back a feed. Your reactions update the wallet.
The room is wiped. The memory rides home.
Temporary working data is discarded. Your updated history goes back in your pocket. Tomorrow, the app personalizes again without keeping a dossier.
Anything readable can be copied
If the room leaks a conclusion like “financially vulnerable,” deleting the raw data changed nothing. The room has to police its outputs, not just promise to forget.
Some things were never only yours
A post you publish, a message someone received, a payment record, a narrow enforcement note. Those need their own rules, and the enforcement note needs the same expiry and review as any other restriction.
Pieces of this already exist. Researchers study on-device recommendation, which keeps personal history on your phone while choosing what to show you. The catch is that simply running a company’s code on your phone isn’t enough, since that code could phone home. A trusted layer, controlled by your operating system or your wallet, has to limit what it can send.
For jobs too heavy for a phone, Apple’s Private Cloud Compute is a concrete precedent for the sealed room: its published design says personal data is used only to fulfill the request, isn’t retained afterward, is shielded from privileged operator access, and runs software that outside researchers can inspect. It isn’t a social network, and it depends on hardware and software working as designed. But it is much stronger than a deletion promise.
And the Solid project separates apps from personal data stores called Pods, where you choose what lives where and who can read it. Its own FAQ names the limit we keep bumping into: once an app has read access, Solid can’t stop it from copying the data. Storage alone isn’t enough. You also have to constrain the computation.
Would a wallet have saved the bakery? Partly. A platform with less of your history has less to classify you with, and you could carry your records, your work, and, ideally, the customers who opted in to hear from you to a competitor. But a ban only needs an account ID and one “restricted” flag. A platform that remembers nothing at all can’t keep out a repeat scammer either. So the wallet still needs fixes one through six for whatever narrow enforcement record survives.
The wallet’s real power: services become replaceable while your continuity stays yours.
Run the bakery through both systems
Same owner, same old account connection, same mistake. Flip the switch to compare.
- 01An old account connection triggers concernConnection and conclusion merge into one verdictRecords the connection and the inference separately
- 02The system suspects immediate harmEverything switches off at oncePauses only the risky capability, with an expiry date
- 03The owner notices months laterDeadline passed: no review, everA fresh look; a missed notice doesn’t settle the merits
- 04The old evidence is inconclusiveSuspicion stays permanentSupervised ads under conditions, expanding with good conduct
- 05The owner disputes itBot replies, or pay for better supportIndependent review, funded by the platforms
- 06The original flag is overturnedEach copied restriction needs its own appealEvery restriction built on it is reconsidered automatically
- 07The platform still refuses serviceRecords and customer list stay behindRecords and opted-in customers come with the owner
Outcome: one flag, no review, no way back, years of customer relationships stranded.Outcome: a pause, a deadline for the pause, a supervised way back, and an exit that keeps the customers.
Notice that none of this needs software alone. A platform that owns the whole system can rewrite its code any time it likes. Durable guarantees need enforceable rules, independent oversight, or services whose governance gives users a real share of control. If I had to start somewhere, I’d test expiring restrictions plus supervised recovery with one willing platform, and measure what matters: days of wrongful lost access, repeat restrictions after a correction, successful recoveries, and harm to other users.
The history of data is a history of rights for the person in the file
Walk back down the hall and look at what actually fixed things, each time.
Not better locks. Locks protected the record for whoever held it. What protected people was a slow accumulation of rights against the holder. The right to see your credit file. The right to correct a government record. Limits on using information for a new purpose. Each one moved a little power from the desk to the person in the drawer.
Automated enforcement is the next desk. It holds a file, makes a judgment, enforces the judgment itself, and recognizes you when you come back. That’s the secret credit report of the 1800s with a new capability bolted on: the writer of the file can also lock the door.
So the next right isn’t about secrecy at all. It’s an answer, in advance, to the question that makes people afraid of these systems in the first place:
“What happens to me if the system gets this wrong?”
You should be able to see what stays available, when the restriction has to be reconsidered, who can override it, and how you keep living while it’s sorted out. Get that right, and the bakery has a bad week instead of a closed business.
Your data is probably safe. The fight now is making sure you are.
This essay began as a conversation with an AI about data, power, and an automated ad ban, and then got checked against sources. The bakery is a composite, not a real business. Meta’s appeal window comes from its help page for suspended or disabled accounts, which says the period varies by region. Other sources: the 1973 HEW report; the FTC’s 2014 data broker study, 2024 X-Mode order, and Ashley Madison settlement; the FBI’s December 2024 generative-AI warning; the U.S. Holocaust Memorial Museum; NIST’s AI RMF; the W3C Verifiable Credentials model; and Apple’s Private Cloud Compute design. The arrow tally, the case file, the control panel, and the replay are my illustrations, not data. Nothing here is legal advice.
If “a way back when the system is wrong” stuck with you
Nobody Will Ever Admit They Were Wrong
The institutional version of this essay’s fix #4: designing organizations that correct their mistakes even when no one in them will concede.
Read the essay →Essay · Content PilotsYour Voice Is Not a Password Anymore
The impersonation drawer, opened all the way: why a cloned voice works, and how to prove who you are when anyone can sound like you.
Read the essay →Essay · Content PilotsAmerica Has 13 Single Points of Failure
What happens when one actor holds too much power over a system, and the engineering that limits the damage no matter who holds the key.
Read the essay →Primary source · FTCData Brokers: A Call for Transparency
The 2014 report on the companies that buy, combine, and infer things about people who have never heard of them.
Read the report →Does one platform hold your entire customer list?
If a single account ban would cut you off from the people who buy from you, you’re running the bakery’s risk. The simplest exit is one you own: your own website, with booking and payments built in, where customers can find you no matter what any platform decides. That’s what I build at Content Pilots. And if you spot something in this essay a privacy lawyer would wince at, tell me. I’ll correct the file.