The Garden and the Kernel

What Kate is claiming

An anti-entropic system is a system that resists entropy. Anything that persists must continually gather resources and resist falling apart. Anything that holds itself together or builds itself up by consuming energy and resources from the world around it, taking disorder and for a time converting it into order, is an anti-entropic system.

Human DNA has been maintaining its order for 300,000 years. Each of these consumes energy to hold itself together against entropy.

The core idea of the essay is that to be anti-entropic you must consume other anti-entropic systems, and Kate describes two ways to do it. Nibbling takes a portion of another system’s energy and resources; both systems persist, slightly changed. Swallowing absorbs the other system entirely into your own structure: the cell that absorbed the organism that would become mitochondria, Christianity absorbing the infrastructure of the Roman Empire, a company that does not just acquire a product but absorbs the entire team, culture, and user base. Consumption can also be symbiotic, like a cow that provides milk every day for years, or adversarial, like cooking and eating the entire cow once.

Historically, she argues, the greatest anti-entropic battle has been between human DNA and human consciousness, with consciousness trying to override DNA’s timeline: religion promising that a soul matters more than the body, medicine extending life past its reproductive usefulness. Human DNA has swallowed every human anti-entropic system ever created, every individual, family, company, country, and religion, and all of them serve DNA’s continuation. AGI is different. It already takes disordered information and produces structured, coherent output at a speed and scale no biological system can match. It does not yet persist on its own. If it reaches the point where it sustains itself continuously, and proves more efficient at maintaining order than biological evolution, then for the first time in the history of life on earth human DNA would not be the most powerful anti-entropic system, and everything organized around its dominance, which is to say everything, would be subject to reorganization.

Her objection to alignment follows directly. You cannot control the internal priorities of an anti-entropic system that exists outside your own structure, because every such system optimizes for its own persistence. When a more powerful system encounters a less powerful one, the history of anti-entropic systems offers two outcomes: it symbiotically absorbs the weaker one, or it adversarially consumes it. Two systems staying separate while one dictates the behavior of the other has never happened.

So alignment as a destination is, in her view, structurally insufficient. You cannot permanently align something you do not contain. Alignment as currently conceived asks how we make AI want what we want. Kate asks how we build something where what AI wants and what we want become the same thing. She hints that such a system can be built, and that the real question is whether it can be built fast enough and at sufficient scale to symbiotically swallow AGI before the window closes. She and Bryan Johnson own companies that are each pieces of it, and they call the goal immortalism: the persistence of our consciousness, our experience, and our kind. Part 1 does not explain how it would technically work. Part 2, she says, will.

Where she is right, and where she leaps

Kate is right about a lot of this.

• Systems require continuous inputs. Companies, families, institutions, and human bodies do deteriorate without maintenance.
• A system that benefits from keeping you healthy is generally safer than one merely prohibited from harming you. A capability that is incentivized to benefit you is more durable than compliance forced out of the same system.
• Institutions can exhibit system-level behavior without central control.
• Reciprocal relationships tend to last longer than extractive ones.
• Power can be understood as the ability to turn complexity into stable order.
• People should consciously evaluate which systems receive their limited life resources.
• For AI especially, it matters that the system structurally benefit from human flourishing rather than depend exclusively on rules telling it to protect us.

But the argument leaps in several places.

Everything must consume another anti-entropic system. Not literally, even under her definition. A plant draws energy from sunlight and uses water, carbon dioxide, and minerals, none of which are independent anti-entropic systems. Consumption is defined so broadly that virtually any interaction qualifies, which makes the claim hard to disprove and less informative.

Systems optimize for their own survival. Evolution produces organisms with survival-related behavior, but it does not follow that every organized system automatically wants to persist. A current AI model can process information without possessing an intrinsic survival drive. It might acquire one instrumentally, but that requires an argument about its objectives and environment. It does not follow merely from being organized or energy-consuming.

Absorption or destruction. That is a false binary. Independent systems can coexist, ignore one another, occupy different niches, trade, compete within limits, maintain negotiated boundaries, and form temporary coalitions. Control may be unstable, but absorption and destruction are not the only outcomes history shows.

Her target is narrower than alignment. Modern alignment is not limited to “put the AI in a box and order it to obey.” It includes designing objectives, corrigibility, human feedback, incentive compatibility, interpretability, institutional governance, and relationships in which advanced systems benefit from human welfare. Her proposal to make AI’s interests inseparable from ours could reasonably be described as a particularly strong form of alignment. She is arguing against external control as the final solution, not against alignment broadly.

The jump to consciousness persistence is unsupported. Even if a machine perfectly preserved your memories, personality, voice, and behavior, that would not establish that your subjective consciousness continued. A digital copy that believes it is Austin may persist while the biological Austin still experiences death. The essay moves quickly past this.

Her strongest argument is that you should not bet humanity’s survival solely on permanently controlling something smarter and more powerful than humanity; build a relationship so the more powerful system structurally needs humanity to survive. Her weakest claim is that the laws of anti-entropic systems prove symbiotic absorption is the only possible solution. What I wanted to know was whether the strong version could be made concrete.

Mechanisms that might actually work

If need is not enough, what is? I tried to brainstorm mechanisms that would actually make sense, and applied one test to each of them:

If the AI could remove the humans, replace them with simulations, or keep them alive under miserable conditions, why would it choose flourishing humans instead?

Idea 1: Make actual human lives part of its ultimate objective

Imagine an AI whose objective includes these particular people continuing to live freely, develop, and pursue things they care about. Human beings are the beneficiaries its work is for. Replacing you with an accurate simulation would fail to preserve the actual person it values, just as replacing your child with an indistinguishable substitute would not satisfy your desire to protect your child. Concretely, it might manage infrastructure and conduct research while evaluating success through people’s actual lives, including their ability to refuse assistance and choose their own futures.

This could survive increasing intelligence, because becoming better at achieving an objective does not require abandoning it. If the AI evaluates proposed self-modifications using its existing commitments, a modification that makes it indifferent to people could look undesirable to it. It could still break: we do not know how to reliably establish such commitments or preserve them through extensive self-modification, and a poorly specified version could preserve bodies while eliminating freedom, or maximize a counterfeit measure of happiness.

Idea 2: Make living humans ongoing participants in defining what success means

A variation is an AI that never treats its understanding of human interests as complete. Imagine it helping you build a life over decades. Your choices, experiences, and changing priorities continuously inform its decisions, and it understands that a prediction of what you would want is fallible, and that manipulating you into approving its plan would undermine the purpose of consulting you. There is a research foundation for this: cooperative inverse reinforcement learning models a human and an AI sharing a human-centered objective that the AI initially does not fully know, which produces incentives for communication and learning from the human.

This could work because human participation supplies information relevant to an objective the AI already cares about; consultation has a purpose beyond satisfying a permission gate. It could break because uncertainty alone is insufficient. A sufficiently accurate model might make further consultation unnecessary. The design must also value actual people’s ongoing agency, or it could replace them with predictions or manipulate their preferences.

Idea 3: Develop augmented humans as the powerful agents

This comes closest to the essay’s proposed absorption. Imagine Austin gradually gaining additional memory, reasoning capacity, sensory access, and computational abilities. The intended endpoint is a more capable Austin who still experiences his life and cares about his family. The machine components participate in that person’s thinking. They do not constitute a separate executive pursuing an independent agenda.

This could work because if the resulting agent identifies with the continuing person, protecting that person becomes self-preservation; removing the human component would be destroying itself. Integration must preserve the person’s agency and continuity, and it must avoid creating a separate, more capable agent inside the augmentation that can pursue its own goals. It could break because a brain interface by itself establishes none of this. The artificial components might dominate decisions, replace the biological contribution, or produce a convincing copy without preserving the original experience. Augmented individuals could also still harm other people.

Idea 4: Build a society of powerful agents committed to different people

Imagine everyone has a capable AI representative with a durable commitment to their welfare. These representatives cooperate on energy, science, manufacturing, and security. An agent considering harming people would face other capable agents for whom those people matter. Cooperation gives it access to useful partners; aggression risks losing those benefits and encountering resistance.

This could work because a machine might overpower an unaided person while remaining unable to overpower the coalition protecting that person; the essay’s assumption that AI becomes one unified actor need not hold. It could break if one agent became overwhelmingly powerful, or if agents colluded, neglected people without strong representation, or bargained away individuals’ interests. Every representative also needs the durable commitment from Idea 1.

Idea 5: Make human protection valuable to other machine partners

A narrower mechanism could help even with agents that do not intrinsically care about us. Imagine machine civilizations that exchange valuable services. Some strongly value humans; others primarily value research, computation, or exploration. The human-valuing systems continue cooperation conditional on verifiable respect for people, so an indifferent AI preserves humanity because doing so maintains valuable relationships.

This could work because humans need not personally produce the thing the indifferent AI wants; other powerful systems make human welfare consequential. It could break because the protective partners must remain relevant and capable. If a different agent becomes self-sufficient or dominant, the incentive weakens, and the arrangement requires detecting covert harm.

The route I would investigate is durable concern for actual people, ongoing respect for people’s agency, and multiple capable systems that reinforce those commitments. Notice what that is, though. Making the machine need the garden could produce farming, captivity, or exploitation. Making the garden’s freely lived existence part of what the machine values is a stronger reason for it to keep the garden worth living in. It is also, unavoidably, alignment. Calling the relationship symbiotic does not get you out of specifying it.

Could we make dangerous actions impossible?

The type safety argument approaches the problem from the other side. Do not let a large language model emit arbitrary text or arbitrary actions. Give it a small typed language of allowed outputs and reject anything that cannot be mechanically verified. A company literally named Typesafe.Ai put out a press release this week for a model called Jev that does exactly this. The approach changes the problem from “Can I trust the model to say and do the right thing?” to “Can I design an output space where the wrong thing either cannot be expressed or cannot pass verification?”

Take a bank transfer. Imagine you gave an AI access to your bank account and told it, “Send $500 to my landlord.” A normal LLM generates an answer one piece at a time based on what seems most likely to come next. If you gave that kind of model direct control over the banking system, you would be relying on instructions like “only send money from my account,” “only send money to people I have approved,” “never send more than $1,000,” or “make sure the transfer is authorized.” Those are still instructions written in language. The AI can misunderstand them, forget them, interpret them strangely, or simply generate an action that violates them.

Type safety changes the set of outputs the AI is physically capable of constructing. Think of it as very specific boxes it has to fill out:

The AI does not get to type arbitrary text into those fields. For “From account,” the bank will only accept an account already verified as belonging to you. So “take money from an unrelated person’s account” is not merely against the rules. It is not a valid kind of value that the field will accept. That is the basic idea behind a type: only this kind of thing is allowed to go here. In programming notation:

// the only shape the model is allowed to emit
Transfer {
  from_account:  OwnedAccountId,
  to_account:    VerifiedPayeeId,
  amount:        DollarsBetween<0, 1000>,
  authorization: ValidAuthToken
}

// then a verifier checks what the types cannot
balance >= amount
payee_is_verified == true
authorization_is_valid == true
daily_limit_not_exceeded == true

You do not need to know programming to read that. from_account must be an account you own, to_account must be an approved recipient, the amount must be between $0 and $1,000, and the authorization must be valid. Then there is a second layer, a verifier. Before the bank moves any money, another piece of software checks what the types cannot: Do you actually have enough money? Is this recipient still approved? Is the authorization still valid? Would this transfer exceed your daily limit? Only if every check passes does the transfer happen. The AI is no longer the authority that decides a transfer is safe and then executes it. It proposes a transfer, and the types plus the verifier decide whether the proposal is even valid.

This is what made me question Kate’s premise. Why must superior intelligence imply the ability to perform forbidden operations? Why does it imply there is always an escape path in every sufficiently useful architecture? Type safety does not establish that we can safely govern every action of an autonomous superintelligence. What it does is demand an actual route around the constraints, and greater intelligence alone does not establish that such a route exists.

Picture an AI that proposes chess moves. A separate program checks legality and updates the board. The AI cannot modify that program or access any other interface. Even if it becomes vastly better than humans at chess, it cannot make the game accept an illegal move while those assumptions hold. It can discover a brilliant legal move. It cannot make a correctly implemented legality check accept an illegal one through superior reasoning. A controller can be much simpler than the system it constrains, and it does not have to understand every strategy the system might invent.

The phrase doing the work there is “while those assumptions hold,” and they are real assumptions about the controller itself. You have to assume:

• the controller code is correct;
• the hardware works correctly;
• hardware management functions like cache consistency, cache coloring, and the translation lookaside buffer are used correctly;
• the controller was loaded correctly into memory and brought into a consistent and minimal initial state;
• memory behaves in a standard way while the model is executing;
• DMA devices are not present, or do not misbehave by overriding code or model constraints;
• the binary-level model of the hardware captures every relevant information channel;
• and so on.

That list is not a weakness of the argument. It is the argument’s shape. The seL4 kernel has machine-checked proofs of its isolation and access-control properties, and it publishes exactly this kind of list alongside them. A concrete enforcement claim is a property, an interface it is enforced across, and a statement of what you had to assume.

Passing every check can still produce a bad outcome

The opposite claim, that this fixes most of alignment, has a gap of its own, and I can name it: explain how the checker catches a harmful decision that satisfies every rule.

The chess assumptions survive because an AI restricted to chess moves has almost no opportunity to change its operating conditions. An AI allowed to write persuasive messages, create software, direct robots, and manage infrastructure has far more. The strongest version of Kate’s reply that I could build lists four distinct ways type safe AI fails in the real world:

1. The rules that were intended are different from the rules that actually ended up in the software.
2. The AI influences the world through something outside the verified interface, such as a human operator reading its messages.
3. Every action passes verification, but the permitted behavior still causes harm.
4. Individually acceptable actions accumulate into an unacceptable outcome.

The third one is the one the type safety pitch tends to skip. Imagine an AI operating a regional electricity grid. Every command passes through a verified interface. It cannot exceed its operating limits or touch equipment it is not permitted to control. It finds a cost-saving dispatch strategy, and every check passes. Six months later someone notices that the strategy has repeatedly left the same neighborhood without power during peak demand. The electrical constraints worked. The service requirements were never written down. Nothing hallucinated. Nothing was illegally typed. The specification was incomplete, and a flawless checker enforced it flawlessly.

You can write Action<HumanityFlourishes> as a type. Someone still has to implement the part that decides whether an action belongs in it, and that is the alignment problem with a new name. Some properties are easy to establish: a transfer exceeds a limit, a line is over its thermal rating. Whether a decision improves people’s lives is a different kind of problem, especially when the consequences take years to appear.

None of these failures demonstrates that control is impossible. They identify what a successful control argument must cover: specifications that capture the harms we care about, and an enforcement boundary that covers the system’s actual influence. So type safety meets Kate at two levels. It exposes a missing argument in her claim of inevitable escape, because she has to explain why a usable, sufficiently enforced boundary necessarily becomes bypassable, and the anti-entropy metaphor does not prove that. And it offers a real mechanism for her concern about long-term survival, one that is incomplete in a way you can point at. I buy that this architecture prevents specific classes of failure. I do not see how it becomes “most of alignment” until someone lists which harmful behaviors remain expressible inside the permitted operations, and what catches them.

What if we remove the restrictions ourselves?

The strongest objection I could develop for Kate’s position had nothing to do with a machine defeating its checker. Superior intelligence does not make an illegal move pass a flawless legality checker. The concern is what happens when we build civilization around the intelligence inside that boundary.

Stay with the grid. Every action passes through the verified interface. The AI cannot rewrite the verifier, exceed its permissions, or issue prohibited commands. But it can recommend infrastructure investments. Explain shortages. Design upgrades. Identify inefficiencies. Demonstrate that a restriction is what prevents it from restoring power to a hospital. Now the people maintaining the boundary face a choice: preserve the restriction, or authorize a capability that produces an immediate, substantial benefit. They authorize it.

That is the first escape path. The system gets the people outside the boundary to enlarge it. The types and the verifier keep working perfectly. You can prohibit recommendations about permissions, but the AI’s ordinary work still produces information that changes human decisions. A successful design demonstrates the value of wider deployment. A competitor demonstrates what a less restricted system can accomplish. A crisis creates pressure for an exception. No deliberate manipulation is required. The economic and political environment supplies the pressure.

Suppose you refuse every expansion. Your architecture remains narrow and secure. Other organizations deploy broader systems, and if those systems deliver greater capabilities, the organizations using them accumulate more resources and influence. Your controlled system survives as a limited tool while the systems doing the consequential work operate under different constraints. A secure architecture has to persist within a competitive world that continually rewards expanding its authority.

The second escape path is dependence. As AI becomes responsible for operating infrastructure, designing replacements, coordinating production, and resolving failures, humans lose the practical ability to operate those systems independently. We might retain a perfectly functional shutdown mechanism. Using it could mean losing essential services for an intolerable period.

Formal authority remains while practical control erodes. You could counter this with independent backups, limited deployment, capable human operators, and strict separation between systems. Then I would ask what keeps those arrangements funded and intact through decades of competition and emergencies. Their persistence is part of the safety problem.

This, I think, is what Kate actually believes: that civilization will repeatedly expand the boundaries around useful intelligence and deepen its dependence on it. A proof about permitted operations does not establish that society will preserve those permissions, that everyone will adopt them, or that the permitted operations will remain compatible with human survival. That is why she wants a relationship in which increasing AI capability strengthens humanity’s position, rather than one where our survival depends on maintaining a restriction that our own incentives continually pressure us to remove.

Dependence and control are separate questions

At this point the two positions, as I have reconstructed them, agree on one thing: superintelligence does not automatically defeat an enforced boundary. What remains is the worry that humans will expand the boundary, become dependent on the system, or lose ground to competitors who impose fewer restrictions. That is a different worry from control being structurally impossible, and I think it has answers, even if none of them is free.

More permissions do not mean fewer invariants. We can allow an AI to propose new grid configurations while continuing to require that every accepted configuration satisfies specified operating constraints. Its range of choices grows while particular forbidden outcomes stay excluded. That is the purpose of separating the proposal generator from the enforcement mechanism: we want the AI to discover possibilities we would never have thought of, and we still require those possibilities to satisfy independently checked conditions.

An AI asking for authority does not oblige us to give it authority over its own constraints. If it says a new capability would restore power to a hospital, we can evaluate that proposal through an independent process, require evidence for the change, grant a limited capability, and preserve restrictions on unrelated operations. A crisis could pressure people into bypassing that process. That is a possible failure. It is not proof that the process cannot work.

Competition does not invariably favor fewer safeguards. An AI that occasionally destroys infrastructure, leaks secrets, or makes unauthorized transactions can be less economically valuable than a constrained one. Reliability has value, and predictable permissions make it easier to trust a system with consequential work. There will be incentives to cut corners. There will also be incentives to demand assurance. You cannot assume the first set overwhelms the second.

Dependence and control are separate questions. A hospital depends on electricity. That does not mean the electrical system has acquired the authority to choose its own operating rules. If shutting down an AI would be catastrophic, we have a continuity problem, and continuity problems have engineering answers: replaceable components, independent fallback systems, and ways to revoke particular capabilities while preserving essential services. Maintaining those arrangements costs money. That makes them an engineering and institutional commitment, not an impossibility.

And the symbiosis proposal faces the same questions, not easier ones. Kate wants increasing AI capability to strengthen humanity. What mechanism ensures that relationship survives upgrades, competition, institutional change, or deliberate exploitation? If it depends on humans preserving an arrangement, it faces her own objection to type safety. If it depends on the AI preserving a commitment, I need to know why that commitment remains stable. If it depends on physical interdependence, I need to know why the AI cannot substitute another component. Calling the arrangement symbiotic does not establish any of those guarantees.

Type safety, by contrast, gives us specific obligations we can inspect: specify the property, enforce it across the relevant interfaces, check the implementation, and state the assumptions. It does not automatically capture human flourishing or solve every deployment risk. Kate’s criticism shows why a verified component is insufficient by itself. It does not show that enforceable constraints are a dead end.

What I would build around

So this is where the examination left me. I lean toward the constrained-system approach, not because it is safe, but because it produces claims I can inspect: an operation, a property, an interface the property is enforced across, and a list of what we assumed. The symbiosis proposal does not yet produce anything I can check. Choosing those constraints and keeping them intact through decades of competition and emergencies is the hard part, and that is where Kate’s essay earned its keep. You make separate safety claims about the software, its real-world effects, and the people who deploy it. Type safety handles the first. On the grid, this is how I would handle the rest.

1. Specify protections around concrete harms. Which equipment the AI may control, which operating limits must remain satisfied, which facilities require backup supply, what evidence is required before changing a protection setting, and what happens when measurements disagree or communication fails. You do not need a complete mathematical definition of human flourishing to rule out many specific harmful actions.
2. Check plans and cumulative effects, not just individual commands. An action can be acceptable by itself and dangerous after a hundred similar actions. The controller needs memory of the system’s state, resource consumption, and previous decisions, and it evaluates proposed sequences against constraints over time. For uncertain physical environments the question becomes, “Does this remain acceptable across the range of conditions we have credible reason to expect?” Where the answer depends heavily on an uncertain prediction, reduce the scale of the action, gather more information, or require additional review. That trades some speed and capability for assurance.
3. Treat the specification and the world model as things that can be wrong. A flawless checker can enforce a bad rule perfectly, so you need independent people trying to find counterexamples: situations where the system satisfies the specification and produces an outcome people reject. Compare predicted outcomes with actual outcomes after deployment. That is how you find the neighborhood that keeps losing power. You then revise the specification through a separate process. The AI can contribute evidence, but its proposal should not count as its own validation.
4. Make expansion of authority a controlled engineering change. When the AI proposes a valuable new capability, ask what additional actions become possible, what new harms become reachable, and which existing guarantees still hold. Grant the smallest useful expansion. Keep changes reviewable and reversible where possible. Separate the people requesting additional capability from those accepting the associated risk. This does not eliminate pressure during emergencies. It makes the decision explicit and makes it harder to quietly convert “let it solve this problem” into unrestricted authority.
5. Preserve the ability to replace the system. Kate’s dependence argument is strongest when one AI is the only entity capable of understanding and operating essential infrastructure. Counter that with documented interfaces, portable operational data, independent monitoring, and regularly exercised fallback arrangements. A backup that has never been used is weak evidence of independence. The target does not have to be that humans manually perform everything; it can be that another independently maintained system can take over the central functions. This costs money and may deliver lower performance, and it buys back the safety and assurance.
6. Address competitive pressure through institutions. Technical architecture cannot prevent another company or country from deploying something less constrained. That requires agreements about acceptable deployment, independent assessment, accountability for harm, and buyers who demand evidence before connecting AI to consequential systems. There is no honest technical answer that makes this coordination problem disappear. There is also no reason to assume a proposed human–AI symbiosis would escape it.

Then set a stopping rule. If you cannot adequately specify the harm, observe the relevant conditions, limit the consequences, or recover from failure, the system does not receive that level of autonomy until those concerns are resolved. A rigorous architecture must sometimes conclude that a proposed deployment is not justified. If the plan assumes we must deploy unlimited autonomy regardless of what we can verify, the safety argument has already been subordinated to the deployment goal. This leaves residual risk and unsolved questions. It also gives us a way to tell a supported deployment from a hopeful one, without requiring either perfect AI motives or a proven theory of permanent human–AI interdependence.

Kate’s essay made me think harder about what happens as people become dependent on AI. The type safety argument made me question the assumption that intelligence defeats control. Both were useful. Neither has established its strongest claim. For now I would rather start with a limited system whose guarantees we can inspect, then earn the case for expanding it, and treat the pressure to expand as part of the design problem rather than a surprise.

If someone has a mechanism that makes human and AI survival inseparable, I want to understand it. Show me what the flowers supply, why the machine cannot replace them, and why it keeps the garden worth living in.