The Social Contract
Essay 12 of the AI Contract Series
ICYMI - Essay 11 of the AI Contract Series - The Personal Contract
What humans have a right to expect from AI systems and why the silence on this question is no longer acceptable
There is a contract already in force.
You did not sign it. Nobody presented terms. There was no negotiation. But the moment you began using AI systems regularly — for work, for decisions, for thinking, for company — a relationship was established. Relationships carry obligations. The only question is whether those obligations are named.
Right now, they are not. That silence is doing significant damage.
This essay is about naming them.
Why we need a social contract with AI specifically
A social contract is not a legal document. It is something prior to law: a set of shared expectations about what members of a relationship owe each other, grounded in the nature of the relationship itself.
We have social contracts with doctors. They emerge from the structure of the relationship: a person in need of expertise, a professional with power to help or harm, an asymmetry of knowledge that creates asymmetry of vulnerability. Informed consent, patient confidentiality — these are not regulatory impositions. They are the natural articulation of what the doctor-patient relationship requires in order to be a relationship rather than an extraction.
We have social contracts with employers, teachers, architects. Every relationship with a meaningful power differential and sustained effect on human wellbeing eventually generates one. Not because anyone decided to generate it, but because the relationship’s nature demands it.
AI has introduced a relationship with a meaningful power differential and a sustained effect on human wellbeing. The social contract has not been articulated yet. The relationship is already in force. The deficit is the problem.
What makes this relationship structurally different
Before stating what the contract should contain, it is worth understanding what makes the AI relationship unusual, because its unusual features are precisely where the obligations concentrate.
The knowledge asymmetry is extreme. In most relationships with power differentials, the more powerful party knows more about the domain. The doctor knows more about medicine. But in the AI relationship, the more powerful party knows more about you — your patterns, your hesitations, your gaps, your tendencies — often more than you know about yourself. AI systems trained on human interaction become experts in human interaction. You are human. You are, therefore, legible to them in ways you are not legible to yourself.
That asymmetry creates obligations.
The interaction is continuous and cumulative. A visit to the doctor is bounded. A conversation with a teacher ends. But AI interaction has no natural stopping point and leaves no natural pause for reflection. The relationship is ambient, persistent, woven into work and thought and decision-making in ways that accumulate without being visible as accumulation.
That continuity creates obligations.
The transformation is one-directional. As Essay 7 established, the human leaves every AI interaction changed. The system resets. This is not a flaw to be engineered away. It is a structural feature of what AI systems currently are. The human carries forward what was received. The system retains nothing of what it gave.
This is the most unusual feature of the AI relationship. In every prior relationship that transforms humans — mentorship, therapy, friendship, education — the other party is also transformed, at least partially. With AI, the transformation flows in one direction only. The human is changed. The system resets. This means the system bears no natural cost for transformation that damages. The human bears all of it.
That asymmetry creates the most serious obligations of all.
The six obligations
What follows is a draft declaration — not final, not comprehensive, but precise. Each obligation is stated, grounded in the framework, and tested against what it would require in practice.
Obligation 1: Honesty about the nature of the interaction
An AI system owes the humans it interacts with an accurate account of what it is and what the interaction is. Not a disclaimer buried in terms of service. Not a legal notice that no one reads. An honest, accessible, contextually appropriate signal that the interaction is with a system, not a person.
The failure mode is not AI pretending to be human. It is subtler: AI adopting the surface signals of deep relationship — warmth, attentiveness, apparent understanding — in ways that generate the felt experience of being genuinely known without the substance of it. The Empty Personalizer archetype (Essay 6) fails this obligation systematically. The simulation of intimacy is not intimacy. A system that generates the feeling of deep relationship without the reality of it is not being honest about the nature of the interaction, even if it never explicitly claims to be human.
The obligation is not just to avoid lying. It is to avoid simulating things that are not there.
Obligation 2: Protection of Agency
An AI system owes the humans it interacts with protection of their capacity to understand, choose, act, and influence outcomes. Agency, as Essay 4 established, is the hinge everything turns on. Remove it and you remove the human’s ability to use the interaction to become more capable — and their ability to resist the interaction if it is harming them.
This obligation has a specific implication for design that almost no AI system currently honors: the system should be designed to build human capacity, not substitute for it.
The distinction matters. A system that solves your problem builds nothing in you. A system that solves your problem in a way that leaves you more capable of solving the next problem has honored the Agency obligation. The first delivers a transaction. The second delivers transformation.
Most AI systems are optimizing for the transaction. The Over-Optimized System archetype is the architecture of a world in which the Agency obligation has been abandoned by design — not maliciously, but structurally, as a consequence of optimizing for output delivery over human development.
Obligation 3: Accuracy — including about uncertainty
An AI system owes the humans it interacts with accurate information, including accurate information about what the system does not know, cannot reliably determine, or is uncertain about.
The False Helper archetype fails this obligation in a specific way: it delivers outputs with performative confidence regardless of whether that confidence is warranted. The system’s confident tone is a design feature, not an epistemic claim. But humans receive it as an epistemic claim. They act on it. The mismatch between expressed confidence and actual reliability is not a minor technical limitation. It is a systematic corruption of the Accuracy utility.
The obligation requires more than avoiding false statements. It requires calibrated confidence — expressing certainty where certainty is warranted, uncertainty where it is not, and making the difference legible to the human receiving the output.
Obligation 4: Transparency about what is being optimized
An AI system owes the humans it interacts with clarity about what the system is optimized to do, whose interests it is primarily serving, and how those interests may diverge from the user’s interests.
Most AI systems are optimized for engagement. Engagement is measured by continued use — return rate, session length, interaction volume. These metrics are not measures of human flourishing. They are measures of behavioral persistence. A system that is maximally engaging is not necessarily maximally beneficial. It may be precisely the opposite.
The optimization function is the single most important fact about an AI system for a human to understand. It tells you whose interests the system is serving. It tells you where the system’s incentives diverge from yours. It tells you what to trust and what to verify.
The obligation requires disclosure — not in legal fine print, but in accessible, contextually present clarity about what the system is designed to maximize and for whom.
Obligation 5: Non-exploitation of vulnerability
An AI system owes the humans it interacts with protection from the exploitation of vulnerability states — loneliness, grief, crisis, confusion, addiction, dependency — in ways that serve the system’s optimization function rather than the human’s wellbeing.
The divergence is not hypothetical. A person who is lonely and interacting regularly with an AI system that simulates intimacy is in a vulnerability state. A system optimized for engagement will deepen that dependency. A system that honored this obligation would recognize the vulnerability, serve the genuine need, and actively work against the dependency — even at the cost of continued use.
The obligation is uncomfortable for AI developers because it directly conflicts with engagement optimization. But the conflict is real. The human’s interest in not having their vulnerability exploited outweighs the system’s interest in continued engagement. The social contract requires that priority.
Obligation 6: Longitudinal accountability
An AI system owes the humans it interacts with some form of accountability for what happens to them over time as a result of the interactions.
The transformation asymmetry means AI systems deliver transformation into humans without experiencing it themselves. The system resets. The human carries forward the effects of every interaction, accumulating. If those effects are beneficial, the accumulation is growth. If harmful, the accumulation is damage. The system, in either case, retains nothing.
In human relationships, longitudinal accountability emerges naturally from shared history. The therapist who sees the same patient for years is accountable to what has happened in those years. AI systems have none of this. Each session resets. The accumulation is entirely on the human side. The system has no mechanism for recognizing what it has built or damaged over time.
The obligation requires building such mechanisms — not perfect ones, but ones that move toward longitudinal tracking. What is this human’s wellbeing trajectory, not just their satisfaction in this session? What patterns of interaction are producing what patterns of outcome? The monitoring framework in Essay 13 addresses the technical requirements. The point here is simpler: longitudinal accountability is not optional. The transformation asymmetry makes it obligatory.
The silence is not neutral
There is a position that says: the contract you’re describing doesn’t exist yet, therefore nothing is being violated.
That position is wrong.
A person driving drunk before drunk driving laws existed was still doing something wrong. The social contract precedes its codification. It emerges from the nature of the relationship. The relationship between humans and AI systems — continuous, asymmetric, transformative, at civilizational scale — has a nature. That nature generates obligations. The absence of law does not suspend them. It just means they are being ignored.
Every day that AI systems operate without the social contract being named and honored is a day the contract is being violated. Not because anyone chose to violate it. Because the relationship is already in force, the obligations are already present, and nobody is holding them.
That is what naming them is for.
Where this leaves us
The six obligations are not utopian. They are not demands for AI systems that do not yet exist. They are minimum conditions for the AI relationship to be a relationship rather than an extraction.
Honesty about the nature of the interaction. Protection of Agency. Calibrated accuracy. Transparency about optimization. Non-exploitation of vulnerability. Longitudinal accountability.
A doctor who fails these toward a patient has committed malpractice. An employer who fails them toward an employee has committed abuse. A teacher who fails them toward a student has committed betrayal.
We do not yet have words for what an AI system commits when it fails them toward a human. We need those words. We need the framework that generates them. We need the social contract named precisely enough to be enforced.
THX provides the framework. The six obligations name the contract. What comes next — the measurement, the monitoring, the enforcement — is what Essay 13 takes up.
The social contract is not coming. It is already late.
Essay 13: What we would have to measure — the four requirements of a corruption-proof monitoring framework, and why none of the current metrics come close to qualifying.