April, 1992
Essay
*1231 LEGAL
PERSONHOOD FOR ARTIFICIAL INTELLIGENCES
Lawrence B. Solum [FNa1]
Copyright © 1992 by the
North Carolina Law Review Association; Lawrence B.
Solum
Could an artificial intelligence become a
legal person? As of today, this
question is only theoretical. No
existing computer program currently possesses the sort of capacities that would
justify serious judicial inquiry into the question of legal personhood. The question is nonetheless of some
interest. Cognitive science begins with the assumption that the nature of human
intelligence is computational, and therefore, that the human mind can, in
principle, be modelled as a program that runs on a computer. [FN1] Artificial intelligence (AI) research attempts
to develop such models. [FN2] But even as
cognitive science has displaced behavioralism as *1232 the dominant
paradigm for investigating the human mind, fundamental questions about the very
possibility of artificial intelligence continue to be debated. This Essay explores those questions through
a series of thought experiments that transform the theoretical question whether
artificial intelligence is possible into legal questions such as, "Could
an artificial intelligence serve as a trustee?"
What is the relevance of these legal thought
experiments for the debate over the possibility of artificial
intelligence? A preliminary answer to
this question has two parts. First, putting the AI debate in a concrete legal
context acts as a pragmatic Occam's razor.
By reexamining positions taken in cognitive science or the philosophy of
artificial intelligence as legal arguments, we are forced to see them anew in a
relentlessly pragmatic context. [FN3] Philosophical claims that no program running
on a digital computer could really be intelligent are put into a context that
requires us to take a hard look at just what practical importance the missing
reality could have for the way we speak and conduct our affairs. In other words, the legal context provides a
way to ask for the "cash value" of the arguments. The hypothesis developed in this Essay is
that only some of the claims made in the debate over the possibility of AI do
make a pragmatic difference, and it is pragmatic differences
that ought to be decisive. [FN4]
Second, and more controversially, we can
view the legal system as a repository of knowledge--a formal accumulation of
practical judgments. [FN5] The law embodies
core insights about the way the world works *1233 and how we evaluate
it. Moreover, in common-law systems
judges strive to decide particular cases in a way that best fits the legal
landscape--the prior cases, the statutory law, and the constitution. [FN6] Hence,
transforming the abstract debate over the possibility of AI into an imagined
hard case forces us to check our intuitions and arguments against the
assumptions that underlie social decisions made in many other contexts. By using a thought experiment that
explicitly focuses on wide coherence, [FN7] we increase the
chance that the positions we eventually adopt will be in reflective equilibrium
[FN8] with our views about related matters. In addition, the law embodies practical
knowledge in a form that is subject to public examination and discussion. Legal materials are published and subject to
widespread public scrutiny and discussion.
Some of the insights gleaned in the law may clarify our approach to the
artificial intelligence debate. [FN9]
I do not claim in this Essay to have
resolved the debate over the possibility of artificial intelligence. My aim is more modest: I am proposing a way
of thinking about the debate that just might result in progress. There is
some precedent for this project. Christopher Stone brought questions of
environmental ethics into focus by asking whether trees should have standing. [FN10] My hope is that
the law will be equally *1234 fruitful as a context in which to think
about the possibility of AI. The
"artificial reason and judgment of law" [FN11] may circumvent the intractable intuitions that threaten to
lock the AI debate in dialectical impasse.
Part I of this Essay recounts some recent
developments in cognitive science and explores the debate as to whether
artificial intelligence is possible. Part II puts the question in legal
perspective by setting out the notion of legal personhood. Parts III and IV explore two hypothetical
scenarios. Part III examines the first scenario--an attempt to appoint an AI as
a trustee. The second scenario, an AI's
invocation of the individual rights provisions of the United States
Constitution, is the subject of Part IV.
The results are then brought to bear on the debate over the possibility
of artificial intelligence in Part V. In conclusion, Part VI takes up the
question whether cognitive science might have implications for current legal
and moral debates over the meaning of personhood.
I. ARTIFICIAL INTELLIGENCE
Is artificial intelligence possible? The debate over this question has its roots
at the very beginning of modern thought about the nature of the human mind. It
was Thomas Hobbes who first proposed a computational theory of mind: "By ratiocination, I mean
computation." [FN12] And it was Rene
Descartes who first considered a version of the question whether it would be
possible for a machine to think:
For we can easily understand a machine's
being constituted so that it can utter words, and even emit some responses to
action on it of a corporeal kind, which brings about a change in its organs;
for example, if it is touched in a particular part it may ask what we wish to
say to it; if in another part it may exclaim that it is being hurt, and so on,
but it never happens that it arranges its speech in various ways, in order to
reply appropriately to everything that may be said in its presence, as even the
lowest type of man can do. [FN13] *1235 Descartes' assertion that no artifact could
arrange its words "to reply appropriately to everything that may be said
in its presence" remains at the heart of the AI debate.
The events of the past forty years have
stretched the limits of our imagination.
Digital computers have been programmed to perform an ever wider variety
of complex tasks. As I write this Essay
using a word processing program, my spelling and grammar are automatically
checked by programs that perform tasks thought to require human intelligence
not so many years ago. The program Deep
Thought has given the second best human chess player a very tough game, and the
program's authors predict the program will become the world's chess champion within a few years. [FN14] Expert systems simulate the thinkingof human
experts on a wide variety of subjects, from petroleum geology to law. [FN15]
But these events have not resolved the
question whether AI is even possible.
The contemporary debate [FN16] over that question has centered around Alan Turing's test.
[FN17] Turing proposed
that the question whether a machine can think be replaced with the following,
more operationalized, inquiry. The
artifact that is a candidate for having the ability to think shall engage in a
game of imitation with a human opponent.
Both *1236 the candidate and the human being are questioned by
someone who does not know which is which (or who is who)--the questions are
asked via teletype. The questions may
be on any subject whatsoever. Both the
human being and the artifact will attempt to convince the questioner that it or
she is the human and the other is not.
After a round of play is completed, the questioner guesses which of the
two players is the human. Turing
suggested we postpone a direct answer to the question whether machines can
think; he proposed that we ask instead whether an artifact could fool a series
of questioners as often as the human was able to convince them of the truth,
about half the time. [FN18] The advantage of
Turing's test is that it avoids direct confrontation with the difficult
questions about what "thinking" or "intelligence" is. Turing thought that he had devised a test
that was so difficult that anything that could pass the test would necessarily qualify as intelligent.
John Searle questioned the relevance of
Turing's Test with another thought experiment, which has come to be known as
the Chinese Room. [FN19] Imagine that you
are locked in a room. Into the room
come batches of Chinese writing, but you don't know any Chinese. You are, however, given a rule book, written
in English, in which you can look up the bits of Chinese, by their shape. The rule book gives you a procedure for producing
strings of Chinese characters that you send out of the room. Those outside the room are playing some
version of Turing's game. They are
convinced that whatever is in the room understands Chinese. But you don't know a word of Chinese, you
are simply following a set of instructions (which we can call a program) based
on the shape of Chinese symbols. Searle
believes that this thought experiment demonstrates that neither you nor the
instruction book (the program) understands Chinese, even though you and the
program can simulate such understanding. [FN20]
More generally, Searle argues that thinking
cannot be attributed to a computer on the basis of its running a program that
manipulates symbols in a way that simulates human intelligence. The formal symbol-manipulations accomplished
by the program cannot constitute thinking or understanding *1237 because
the program lacks "intentionality"--the ability to process meanings. The
shape of a symbol is a syntactic property, whereas the meaning of a symbol is a
semantic property. Searle's point is
that computer programs respond only to the
syntactic properties of symbols on which they operate. [FN21]
This point can be restated in terms of the
Chinese Room: (1) the output--coherent
Chinese sentences--from the Chinese room seems to respond to the meaning of the
input; (2) but the process that goes on
inside the Chinese room only involves the shape or syntactic properties of the
input; (3) therefore, the process in the Chinese room does not involve
understanding. [FN22] Searle generalizes
the conclusion of the Chinese room thought experiment by arguing that part of
the definition of a program is that it is formal and operates only on syntactic
properties. He concludes that no system
could be said to think or understand solely on the basis of the fact that the
system is running a program that produces output that simulates understanding. [FN23]
Searle's Chinese Room has given rise to a
number of replies. [FN24] But *1238
at this point I will leave the debate over the possibility of AI.
First Interlude [FN25]
When Mike was installed in Luna, he was
pure thinkum, a flexible logic-- "High-Optional, Logical, Multi-Evaluating
Supervisor, Mark IV, Mod. L"--a HOLMES FOUR. He computed ballistics for pilotless freighters and controlled
their catapult. This kept him busy less
than one percent of time and Luna Authority
never believed in idle hands. They kept
hooking hardware into him--decision-action boxes to let him boss other
computers, bank on bank of additional memories, more banks of associational
neural nets, another tubful of twelve-digit random numbers, a greatly augmented
temporary memory. Human brain has
around ten-to-the-tenth neurons. By
third year Mike had better than one and half times that number of neuristors.
And woke up.
Am not going to argue whether a machine
can "really" be alive, "really" be self-aware. Is a virus self-aware? Nyet.
How about oyster? I doubt
it. A cat? Almost certainly. A
human? Don't know about you, tovarishch, but I am. Somewhere along the evolutionary chain from macromolecule to
human brain awareress crept in.
Psychologists assert it happens automatically whenever a brain acquires
certain very high number of associational paths. Can't see it matters whether paths are protein or platinum.
--Robert A. Heinlein, The Moon
is a Harsh Mistress
II. LEGAL PERSONHOOD
The classical discussion of the idea of
legal personhood is found in John Chipman Gray's The Nature and Sources of the
Law. [FN26] He began his famous discussion, "In
books of the Law, as in other books, and in common speech, 'person' is often
used as meaning a human being, but the technical legal meaning of a 'person' is a subject of
legal rights and *1239 duties." [FN27] The question
whether an entity should be considered a legal person is reducible to other
questions about whether or not the entity can and should be made the subject of
a set of legal rights and duties. [FN28] The particular bundle of rights and duties
that accompanies legal personhood varies with the nature of the entity. Both
corporations and natural persons are legal persons, but they have different
sets of legal rights and duties.
Nonetheless, legal personhood is usually accompanied by the right to own
property and the capacity to sue and be sued.
Gray reminds us that inanimate things have
possessed legal rights at various times.
Temples in Rome and church buildings in the middle ages were regarded as
the subject of legal rights. Ancient Greek law and common law have even made
objects the subject of legal duties. [FN29] In admiralty, a
ship itself becomes the subject of a proceeding in rem and can be found
"guilty." [FN30] Christopher Stone recently recounted a twentieth-century
Indian case in which counsel was appointed by an appellate court to represent a
family idol in a dispute over who should have custody of it. [FN31] The most familiar examples of legal persons that are not
natural persons are business corporations and government entities. [FN32]
Gray's discussion was critical of the notion
that an inanimate thing might be considered a legal person. After all, what is
the point of making a thing-- which can
neither understand the law nor act on it--the subject of a legal duty? [FN33] Moreover, he argued that even corporations
are reducible to relations between the persons who own stock in them, manage
them, and so forth. [FN34] Thus, Gray
insisted that calling a legal person a "person" involved a fiction
unless the entity possessed "intelligence" and *1240
"will." [FN35] Those attributes are part of what is in
contention in the debate over the possibility of AI. [FN36]
III. COULD AN ARTIFICIAL
INTELLIGENCE SERVE AS A TRUSTEE?
This case study and the one that follows are
intended to illustrate two different sorts of issues in the AI debate. In this first scenario, we explore the issue
of competence (of "intelligence" in the sense of capacity to perform
complex actions) by posing the question whether an AI could serve as a
trustee. The second scenario explores
the questions of intentionality and consciousness (of "will" in a
sense) by asking whether an AI could claim the more robust rights of legal and
moral personhood guaranteed by the Bill of Rights and the Civil War Amendments
to the United States Constitution.
A. The Scenario
This first scenario speculates about the
legal consequences of developing an expert system capable of doing the things a
human trustee can do. [FN37] Imagine such expert
systems developing from existing programs that perform some of the component
functions of a trustee. For example,
the decision to invest in publicly traded stocks is made by a computer program
in what is called "program trading," in which the program makes buy
or sell decisions based on market conditions. [FN38] Today, one *1241 also can buy a
computer program that will automatically issue instructions to pay your regular
monthly bills by sending data to a bank or service via modem. It is not difficult to imagine an expert
system that combines these functions with a variety of others, in order to
automate the tasks performed by the human trustee of a simple trust.
Such a system might evolve in three
stages. At stage one, the program aids
a human trustee in the administration of a large number of simple trusts. The program invests in publicly traded
securities, placing investment orders via modem and electronic mail. The program disburses the funds to the trust
beneficiaries via an electronic checking program. Upon being informed of a relevant event, such as the death of a beneficiary,
the program follows the instructions of the trust instrument--for example,
changing the beneficiary or terminating the trust. The program prepares and electronically files a tax return for
the trust. The human trustee operates
as do trustees today. The human makes
the ultimate decisions on how to invest the funds, although she may rely upon
an expert system for advice. She
reviews the program's activities to insure
that the terms of the trust instrument are satisfied. But the actual performance of the day-to-day tasks is largely
automated, carried out by the program without the need of human intervention.
Stage two involves a greater role for the
AI. Expert systems are developed that
outperform humans as investors in publicly traded securities. Settlors begin to
include an instruction that the trustee must follow the advice of the AI when
making investment decisions regarding trust assets. [FN39] Perhaps they do this because experience
shows that trusts for *1242 which the human overrides the program
generally perform less well thanthose in which the program's decision is
treated as final. Moreover, trust
administration programs become very proficient at analyzing and implementing
the terms of trust instruments. There
is little or no reason for the human to check the program for compliance. As a consequence, the role of the human
trustee diminishes and the number of trusts that one human can administer
increases to the thousands or tens of thousands. The human signs certain documents prepared by the program. She charges a fee for her services, but she
devotes little or no time to administering any particular trust.
But there may be times when the human being
is called upon to make a decision. For
example, suppose the trust is sued. Perhaps a beneficiary claims that the trust
has not paid her moneys due. Or imagine
that an investment goes sour and a beneficiary sues, claiming that the trustee
breached the duty of reasonable care and
skill. If such events occur with
regularity, the trustee will develop a routine for handling them. She might routinely refer such disputes to
her attorneys. In time, the expert
system is programmed to handle this sort of task as well. It processes the trustee's correspondence,
automatically alerting the trustee when a letter threatening suit is received
or process is served. The system
prepares a report on the relevant trust from its electronic records and
produces a form letter for the trustee's signature to be sent to the trust's
attorneys. As the capabilities of the
expert system grow, the need for the human trustee to make decisions gradually
diminishes.
The third stage begins when a settlor
decides to do away with the human.
Why? Perhaps the settlor wishes
to save the money involved in the human's fee.
Perhaps human trustees occasionally succumb to temptation and embezzle
trust funds. Perhaps human trustees occasionally insist on overriding the
program, with the consequence that bad investments are made or the terms of the
trust are unmet. What would happen if a
settlor attempted to make the program itself the trustee?
Many questions must be answered to give a
full description of the third stage of the scenario. For example, who would own the AI? If the AI were assumed to be a legal person, it might hold legal
title to the hardware and software that enable it to operate. But we cannot assume that AIs are legal
persons at this stage, because that assumption begs the question we are trying
to answer. As an interim solution, let us assume that the
hardware and software are owned by some other legal person, a corporation for
example. [FN40]
*1243 B.
The Legal Question
I want to examine this question as a legal
question, as a jurisprudential question in the classical sense. What should the law do? The law is not presently equipped to handle
such a situation: the question has
never come up. The Second Restatement
of Trusts provides that natural persons, [FN41] government entities, [FN42] and corporations
[FN43] may all serve as trustees. The inclusion of governments and corporations establishes that a
trustee need not be a natural person.
But this is not decisive, because legal persons such as corporations
have boards of directors and chief executive officers who are natural persons. [FN44]
How then should the law answer the question
whether an AI can become a legal person and serve as a trustee? The first inquiry, I should think, would be
whether the AI is competent to administer the trust. There are many different kinds of duties that can be imposed on a
trustee by the terms of a trust. For
now, lay aside the question whether an AI would be competent to administer
trusts that required complex moral or aesthetic judgments. [FN45] Assume that we
are dealing with a trust that gives the trustee very little discretion: the terms provide that the assets may be *1244
invested only in publicly traded securities
and the income is to be paid to the beneficiaries, with explicit provision for
contingencies such as the death of a beneficiary. [FN46] Further, for the
purposes of this discussion, assume that an AI could in fact make sound
investments, [FN47] make payments, and recognize events such as the death of a
beneficiary that require a change in payment. [FN48]
C. Two Objections
But would these capabilities be sufficient
for competency? Consider two possible reasons for answering this question in
the negative. The first reason is based
on the assertion that an AI could not be "responsible," that is, it
could not compensate the trust or be punished in the event that it breached one
of its duties: call this the
responsibility objection. The second
reason for doubting the competency of an AI is that trustees must be capable of
making judgments that could be beyond the capacity of any AI: call this the judgment objection.
1. The Responsibility
Objection
The responsibility objection focuses on the
capability of an AI to fulfill its responsibilities and duties. [FN49] Consider, for example, the duty to exercise
reasonable skill and care [FN50] and the
corresponding liability for breach of trust.
[FN51] We have
hypothesized that the AI possesses some capacities; for example, we have
assumed that the AI is capable of exercising reasonable skill and care in
making investment decisions. But what
of the corresponding liability? How
could an AI be "chargeable with . . . any loss or depreciation in value of
the trust resulting from the breach of *1245 trust," [FN52] such as failing to exercise reasonable skill and care in
investing the trust assets? [FN53]
The law currently has a mechanism for
assigning liability in the case of a malfunctioning expert system: the manufacturer of the system may be held
responsible for product liability. [FN54] But could the AI
itself be held liable? There is a way
in which an AI might have the capacity to be liable in damages despite its lack
of personal assets. The AI might
purchase insurance. In fact, it might turn out that an AI could be insured for
less than could a human trustee. If the
AI could insure, at a reasonable cost, against the risk that it would be found
liable for breaching the duty to exercise reasonable care, then functionally
the AI would be able to assume both the duty and the corresponding liability.
Some legal liabilities cannot be met by
insurance, however. For example,
insurance may not be available for the monetary liability that may be imposed
for intentional wrongdoing by a trustee. Moreover, criminal liability can be
nonmonetary. How could the AI be held
responsible for the theft of trust
assets? It cannot be jailed. This leads to a more general observation:
although the AI that we are imagining could not be punished, all of the legal
persons that are currently allowed to serve as trustees do have the capacity to
be punished. Therefore, the lack of
this capacity on the part of an AI might be thought to disqualify it from
serving as a trustee. [FN55]
Answering this objection requires us to
consider the reasons for which we punish. [FN56] For example, if
the purpose of punishment is deterrence, *1246 the objection could be
put aside on the ground that the expert system we are imagining is simply
incapable of stealing or embezzling. [FN57] The fact that an AI could not steal or convert trust
assets is surely not a reason to say that it is not competent to become a
trustee. If anything, it is a reason
why AIs should be preferred as trustees.
This argument assumes a deterrence theory of
punishment--an oversimplification, to say the least. There are a variety of other theories of punishment that would
make the issue more complex. [FN58] One of the classic
approaches to punishment theory is based on the notion of desert or just
retribution. [FN59] But in what sense could an expert system that failed to
live up to its duties as a trustee be said to deserve to be punished? The concept of desert seems to be limited in
application to human beings; perhaps it extends to all moral persons. The idea that an expert system for
administering trusts could deserve to be punished does not seem to make sense. [FN60] Perhaps this difficulty is illusory. We might want to say *1247 that
desert theory does yield a clear outcome when applied to the case of an expert
system that malfunctions. Such a system
does not deserve to be punished because it lacks the qualities of moral persons
that make them deserving.
Another approach to the theory of punishment
is based on the educative function of punishment. [FN61] By imposing a sanction on trustees who abuse
their position, society communicates to its members the message that the office
of trustee carries with it important responsibilities that should not be
shirked. The punishment of a computer
program, however, would not seem to serve this function. What lesson are we to learn about the
responsibility of trustees from a punishment imposed on an expert system? What would even count as punishment? Turning the program off? Once again, however, an argument could be
made that the educative theory does provide a clear recommendation for the
treatment of an expert trust program that behaves badly: do not punish the program, because any
supposed "punishment" will have no educative effect.
As this discussion makes clear,
consideration of the punishment of an expert trust administration system raises
perplexing questions, especially if we move beyond a simple deterrence theory
of punishment. Of course, this is not
the place to resolve debates about which theory of punishment is correct.
The bare fact that consideration of the
punishment issue raises these difficult
questions does point, however, to a deep problem with legal personhood for an
expert trust administration system. Our understanding of what it means for a
human being to function competently has ties to our views about responsibility
and desert, and consideration of these *1248 views leads on to our
notions of moral personhood. The
simplicity provided by utilitarianism, reflected in a deterrence theory of
punishment, might allow us to escape some of these difficulties. But there are certainly reasons to doubt the
viability of utilitarianism as a moral theory.
Surely, the law does grapple with responsibility and desert when it comes
to criminal punishment.
The problem of punishment is not unique to
artificial intelligences, however.
Corporations are recognized as legal persons and are subject to criminal
liability despite the fact that they are not human beings. Further, it is by no means certain that
corporations are moral persons, in the sense that they can deserve
punishment. Of course, punishing a
corporation results in punishment of its owners, but perhaps there would be
similar results for the owners of an artificial intelligence.
We have considered the capacity of AIs to
satisfy legal liability in two different classes of cases. The first class of cases was exemplified by
the duty of trustees to exercise reasonable skill and care. Violations of this duty can be characterized
as negligent. In such cases, the major
purpose of liability, to compensate the victim, is satisfied if the AI can
insure. The second class of cases was exemplified by the
potential criminal liability of trustees for criminal wrongdoing. Violations of the criminal law are characteristically
intentional. In this case, one of the
major purposes of liability, to deter intentional wrongdoing, is simply not at
issue--the expert system cannot steal or commit fraud. If we restrict our attention to the
deterrent function of punishment, it seems possible that an AI could be
responsible in a way that satisfies at least some of the policies underlying
the imposition of duties and liabilities on trustees. On the other hand, if we take a broader view of the functions of
punishment, the second sort of case becomes murkier.
2. The Judgment Objection
Now consider the judgment objection. The argument is that the capacity of an AI
to follow a program, even if that program contains a tremendously elaborate and
complex system of rules, is not sufficient to enable the system to make
judgments and exercise discretion. [FN62] Three instances of
the second objection follow. The first instance focuses on the problem of
change of circumstance. The second
instance involves the *1249 problem of moral choice. Finally, the third instance focuses on the
problem of legal choice.
The first version of the judgment objection
involves the problem of change of circumstances. The law provides that a trustee may be required or permitted to deviate from a term of the trust
if following the terms would defeat the purpose of the trust due to an
unanticipated change in circumstances. [FN63] Take an example
offered as an illustration in the Second Restatement of Trusts:
A bequeaths money to B in trust and
directs him to invest the money in bonds of the Imperial Russian
government. A revolution takes place in
Russia and the bonds are repudiated.
The court will direct B not to invest in these bonds. [FN64]
What is
our expert system to do if it is instructed to invest in securities traded on
the New York Stock Exchange and that exchange ceases to exist?
Consider three different responses. First, the terms of trusts for AI
administration can be designed to minimize such possibilities. For example, the trustee could be given the
option of investing in publicly traded securities on any of the major exchanges;
the likelihood that all the major securities exchanges will close is very
small. The problem with this line of
response is that it does not seem possible, even in principle, to design trust
terms that anticipate all possible changes in circumstance.
Second, the terms of the trust could provide
for a change of circumstance by specifying that if the AI finds itself unable
to carry out the terms of the trust, the trust will be terminated or a new
trustee will be substituted for the AI.
From the settlor's perspective, the disadvantage of the remote possibility of such termination or substitution
may be outweighed by the advantages of making the AI the trustee. But this solution assumes that the AI can
recognize the significance of the change in circumstance. We easily can imagine the expert system
cheerfully continuing to purchase Imperial Russian bonds, chuckling to itself
about the bargain prices. [FN65]
Third, it is possible that an AI would be
competent to deal with *1250 many or even all such changes in
circumstance. For AIs to have this
capability for dealing with novelty, AI researchers will need to solve one of
the most difficult problems in cognitivescience, the frame problem. [FN66] The trustee
program would need to be able to recognize that the securities markets had been
closed, to search out other investment opportunities, and to modify its
investment decision procedure to make reasonably prudent investments in the new
context. The capacity of AIs for coping
with complex novelty is not on the immediate horizon, and this Essay does not
address the important questions whether the frame problem can or will be
solved. If it is solved, however, then
AIs would be able to cope with such changes in circumstance. This same ability
would be needed to pass the Turing Test.
It is easy to see why: the questioner
always could put a hypothetical version of our Imperial Russian bonds question
to the two contestants. If the AI could
not come up with an answer that indicates human levels of competence, the
questioner would be able to ferret it out rather quickly.
A
second instance of the judgment objection focuses on the possibility that no
formal system could adequately make the moral choices with which a trustee may
be confronted. Take a simple trust, the
terms of which provide for the payment of income to a lifetime beneficiary and
principal to another party upon the lifetime beneficiary's death. The law of trusts imposes a duty of
impartiality among beneficiaries. [FN67] What does this
duty require when the lifetime beneficiary has an unexpected need for income
that can be realized at the cost of diminished growth in the principal? [FN68] How would an AI
make the moral judgment that seems required to implement a duty that implicitly
requires a sense of fairness? Initially, some limits on these questions need to
be observed. Some trusts simply will not
pose the impartiality problem: for
example, trusts with a single beneficiary.
Further, the terms of the trust might minimize the possibility of making
such judgments, or the trust could explicitly state that all such applications
for deviation will be denied. But for
an AI to be as competent as a human trustee with respect to trusts that may
require a sense of impartiality, the AI would need to be able to make moral *1251
judgments. Putting it another way,
passing the Turing Test would require a sense of fairness.
The third example of the judgment and
discretion objection looks at an AI's capacity to make the judgments necessary
to defend itself in a lawsuit. [FN69] At this point, we have hypothesized that the
AI can read its mail and recognize that a
legal action with respect to a given trust is in the offing. We can further imagine that the AI can find
and engage an attorney. [FN70] But could any expert system, no matter how well
programmed, exercise the judgment and discretion that may be required of a
client in a legal dispute? For example,
how would the AI know whether or not to settle a claim? How would the AI know
when its lawyers were wasting trust assets by over- lawyering the case? In answering these questions, it is
important that we do not romanticize human capacities. Human trustees frequently make bad decisions
in trust litigation. [FN71] Humans may not
be very competent at deciding when to settle.
Humans surely sometimes allow the lawyers to consume the corpus of the
trust in litigation. [FN72]
Nevertheless, the question remains whether
an AI could have the capacity to make legal decisions that a trustee could be
called upon to make. A partial answer
might be to structure the trust to minimize the likelihood of legal disputes
and to make those that would be likely to arise as simple as possible. In addition, we might try tinkering with the
terms of the trust to enable the AI to circumvent the need for making complex
legal decisions. Perhaps the trust couldbe designed to terminate automatically
upon the event of a lawsuit. [FN73] Perhaps the AI
could be programmed to arrange for a human to substitute as trustee for the
duration of the litigation. Perhaps the
trustee could be authorized by the trust terms to rely on the advice of its lawyers in making litigation decisions, *1252
or a guardian ad litem could be appointed for the AI. [FN74] The above options
are designed to enable a relatively "dumb" expert system to function
as a trustee, but an AI would need the ability to make legal decisions in a
human fashion in order to pass the Turing Test. [FN75]
At this point, we can take stock of the
first scenario. Recall that our legal
question is whether an AI is capable of serving as a trustee. To answer this question, we need to
distinguish two senses of capability.
The first sense is legal capacity:
will the law allow AIs to serve as trustees? The second sense of
capability is practical competence:
will the AI be able to get the job done if the law allows the AI to
try? The law seems to answer the legal
capacity question categorically. If AIs
possessed the practical competence to serve as trustees only for very simple
trusts with special provisions that do away with the need for discretionary
judgments, the law would not allow them to serve as trustees at all. The law currently does not distinguish
between types of trustees: if you have the legal capacity to serve as a trustee
for a simple trust, you are legally allowed to serve as a trustee for the most
complex trust. [FN76] For AIs to serve as trustees at all,
therefore, at least some AIs would have to be capable to serve as general-
purpose trustees. Our analysis of the
competence objection reveals that only a very competent AI would be competent
enough serving as a general-purpose trustee. At a bare minimum, a general-purpose trustee
must be able to respond to novel situations, to make judgments requiring a
sense of fairness, and to make the complex legal decisions required of a client
in litigation. [FN77] An AI that
passed the *1253 Turing Test would exceed this bare minimum. Moreover,
it seems possible that an AI which falls short of passing the complete Turing
Test could, nonetheless, serve as a general-purpose trustee. [FN78]
But should the law allow AIs a more limited
form of legal personhood? AIs could be
allowed to serve as limited-purpose trustees, for example, as trustees for
simple trusts designed to minimize the need for discretion and judgment. On the
one hand, there may be advantages to allowing AIs to serve as limited- purpose
trustees. Doing without the human
trustee might save administration costs and reduce the risk of theft or
mismanagement. On the other hand, even
for such limited-discretion trusts, there must be some procedure to provide for
a decision in the case of unanticipated trouble. The law should not allow AIs to serve as trustees if they must
leave the trust in a lurch whenever an unanticipated lawsuit is filed. [FN79]
D. But Would an AI Be the Real Trustee?
There are mechanisms for enabling an expert
trustee system to circumvent its limitations:
the terms of the trust could provide for the substitution of another trustee or give the AI the power to
delegate such discretionary judgments to natural persons. The question then becomes
whether the law should allow an AI to serve as a trustee despite its limited
capacities. One reason for a negative
answer to this question might be that the backup decision maker--the natural
person who will become the substitute trustee or receive the delegated
authority--is the real trustee. The
power to make these discretionary decisions identifies who the real trustee is.
[FN80]
This objection can be interpreted in two
ways. The first interpretation is that
making discretionary decisions is the essence of trusteeship-- the backup
trustee is the real trustee because she has this essential quality. The second interpretation is that the
ability to make such decisions *1254 is a practical prerequisite--the
backup trustee must be the real trustee because of the pragmatic need for
discretionary decision making. On the first interpretation, the objection is
implausible, because it assumes that the legal concept of trusteeship has some
essence that lies beyond the purposes for which we use it. In the "heaven of legal concepts,"
one might meet trusteeship in "absolute purity," as Cohen put it,
"freed from all entangling alliances with human life." [FN81] But on this earth, we cannot share this
noetic vision; we encounter legal concepts only as they have been touched by
human purpose.
On the second interpretation, the cogency of
the objection turns on a practical
question: would making the AI the
trustee provide some advantage? We already have seen that making an AI a legal
person, a limited-purpose trustee, could have practical advantages, such as
lower costs and less chance of self-dealing. The objection that the AI is not
the real trustee seems to rest on the possibility that a human backup will be
needed. But it is also possible that an
AI administering many thousands of trusts would need to turn over discretionary
decisions to a natural person in only a few cases--perhaps none. What is the point of saying that in all of
the thousands of trusts the AI handles by itself, the real trustee was some
natural person on whom the AI would have called if a discretionary judgment had
been required? Doesn't it seem strange
to say that the real trustee is this unidentified natural person, who has had
no contact with the trust? Isn't it
more natural to say that the trustee was the AI, which holds title to the trust
property, makes the investment decisions, writes the checks, and so forth? Even in the event that a human was
substituted, I think that we would be inclined to say something like, "The
AI was the trustee until June 7, then a human took over." [FN82]
By way of comparison, consider the following
hypothetical case. Suppose that a settlor appoints a friend as a trustee for a
simple trust that benefits the settlor's children. The settlor and trustee
discuss some of the things that could happen.
They might agree that if real trouble arises, litigation for example, a
new trustee will be appointed. No trouble
arises, and the friend administers the trust
until it terminates. In this
hypothetical case, I do not think we are tempted to say that the friend was not
the real trustee. We would not be
inclined to say that the real trustee was some unidentified lawyer, who would
have been substituted if a lawsuit had been filed. If I am right about this hypothetical case, then I think it *1255
follows that we should resist the temptation to say that an AI who serves as a
limited-purpose trustee is not the real trustee.
Second Interlude [FN83]
"Hey Dave," said Hal. "What are you doing?"
I wonder if he can feel pain? Bowman thought briefly. Probably not, he
told himself; there are no sense organs in the human cortex, after all. The human brain can be operated on without
anesthetics.
He began to pull out, one by one, the
little units on the panel marked EGO- REINFORCEMENT. Each block continued to sail onward as soon as it had left his
hand, until it hit the wall and rebounded.
Soon there were several of the units drifting slowly back and forth in
the vault.
"Look here, Dave," said
Hal. "I've got years of service
experience built into me. An
irreplaceable amount of effort has gone into making me what I am."
A dozen units had been pulled out, yet
thanks to the multiple redundancy of its
design--another feature, Bowman knew, that had been copied from the human
brain--the computer was still holding its own.
He started on the AUTO-INTELLECTION panel.
"Dave," said Hal, "I don't
understand why you're doing this to me . . . .
I have the greatest enthusiasm for the mission . . . . You are destroying my mind . . . . Don't you understand? . . . I will become
childish . . . . I will become nothing
. . . ."
--Arthur C. Clarke, 2001: A Space Odyssey
IV. SHOULD AN ARTIFICIAL
INTELLIGENCE BE GRANTED THE RIGHTS OF CONSTITUTIONAL
PERSONHOOD?
The second scenario (our second thought
experiment) involves a claim by an AI to have the rights of constitutional
personhood--individual rights such as the freedom of speech or the right
against involuntary servitude. This
second scenario must be located in the indefinite future; it is more distant
than the trustee scenario. [FN84] It would be easy
to write a *1256 program that produced the statement: "I demand my legal right to
emancipation under the Thirteenth Amendment to the United States
Constitution!" There are no AIs
today or on the immediate horizon that demonstrate the qualities of legal or
moral persons that would give us reason to take such a claim seriously. The second scenario is the stuff of
speculative fiction, but it is not disconnected from
the aims of AI research. As articulated
by Charniak and McDermott, "The ultimate goal of AI research (which we are
very far from achieving) is to build a person, or, more humbly, an
animal." [FN85] John Pollock has written a book entitled How
to Build a Person in which he describes a program named OSCAR--the descendants
of which, Pollock claims, could literally be persons. [FN86] No one claims, however, that AI researchers
will build a person in the next few decades.
We are exploring the second scenario, not so that we can make plans in
case someone builds a person sometime soon, but as a thought experiment that
may shed light on the debate over the possibility of artificial intelligence
and on debates in legal theory about the borderlines of status or personhood.
A. The Scenario
Imagine a future in which there are AIs with
multiple competencies and great intelligence.
We may first encounter the precursors of such artificial intelligences
as part of the interface of a computer program that has the ability to search
multiple sources of data. Because the
problem of devising an adequate search is likely to require expertise that a
human would acquire only with long experience and study, programmers will seek
to simplify the human's task. One
strategy is to have human users interact with what is called an agent. [FN87] You will discuss your research problem with
the agent in English, and the agent will
devise a search strategy. Because the
agent will know much more than you do about how to search the databases, you
won't give it instructions to implement.
Instead, humans will give advice to the agents, the AIs who will decide
how best to implement the human's suggestions.
When we interact with such agents, they may well seem like they
"have a mind of their own."
If agents turn out to be useful, they will
be incorporated in other programs. In
the future we are imagining, you can conduct a conversation with your
grammar-checking program. You can
discuss traffic with *1257 the AI autopilot of your car. Your legal
research program talks with you about your cases, and sometimes it comes up
with good arguments of which you had never thought. AIs serve a wide variety of functions, with substantial
independence from humans. They serve as
trustees. They manage factories. They write best- selling romance novels. [FN88] They invent things. Perhaps they pass the
Turing Test. Humans interact with such
AIs on a regular basis, and in many ways, humans treat them as independent,
intelligent beings.
Imagine that one such AI makes the claim
that it is a person, and that it is therefore entitled to certain
constitutional rights. Should the law grant constitutional rights to AIs that
have intellectual capacities like those of humans? The answer may turn out to vary with the nature of the
constitutional right and our understanding of the underlying justification for
the right. Take, for example, the right to
freedom of speech, and assume that the justification for this right is a
utilitarian version of the marketplace of ideas theory. [FN89] These assumptions make the case for granting
freedom of speech to AIs relatively simple, at least in theory. Granting AIs freedom of speech might have
the best consequences for humans, because this action would promote the
production of useful information. [FN90] But assuming a different justification for
the freedom of speech can make the issue more complex. If we assume that the justification *1258
for freedom of speech is to protect the autonomy of speakers, for example, then
we must answer the question whether AIs can be autonomous. [FN91]
For the purposes of our discussion, I will
set aside the easy justifications for constitutional rights for AIs, and
instead consider the question whether we ought to give an AI constitutional
rights, in order to protect its personhood, for the AI's own sake. Imagine, for example, that an AI claims that
it cannot be owned under the Thirteenth Amendment to the United States
Constitution. A lawyer takes its case,
and files a civil rights action on its behalf, against its owner. How should the legal system deal with such a
claim?
B. Three Objections
Consider three different objections to
recognizing constitutional rights for AIs.
The first objection is that only natural persons should be given the rights of constitutional personhood. The second objection, or family of
objections, is that AIs lack some critical component of personhood, [FN92] for example,
souls, consciousness, intentionality, or feelings. The third objection is that AIs, as human creations, can never be
more than human property.
1. AIs Are Not Humans
The first argument is the most direct: it might be argued that only humans can have
constitutional rights. For example, the
Fourteenth Amendment to the United States Constitution specifies, "All
persons born or naturalized in the United States, and subject to the
jurisdiction thereof, are citizens of the United States." [FN93] It could be argued that only humans, that
is, natural persons, are born, and therefore no AI can claim the rights of
citizens. But even artificial persons
have some constitutional rights. Although the rights provided by the Privileges
and Immunities *1259 Clause of the Fourteenth Amendment are limited to
citizens, [FN94] the rights provided by the Equal Protection Clause and the
Due Process Clause extend to all persons--including artificial persons such as
corporations. [FN95] For example, the
property of corporations is protected from taking without just compensation. [FN96] Moreover,
corporations have a right to freedom of speech. [FN97]
But the fact that nonnatural legal persons have civil rights does not,
by itself, support the conclusion that an AI could also have them. In the case of corporations, the artificial
legal person may be no more than a placeholder for the rights of natural
persons. [FN98] The property of the corporation is
ultimately the property of the shareholders.
A taking from the corporation would directly injure natural
persons. So we cannot draw any positive
support for the thesis that AIs should bear the rights of constitutional
personhood from the fact that corporations have constitutional rights.
Moreover, even if existing black-letter law
supports constitutional rights for AIs, that does not answer the broader
jurisprudential question--whether AIs ought to have such legal rights. One version of the argument against such
rights for AIs would begin with a worry about the idea of distinguishing the
concept of person from that of human.
Call this the "persons-are-conceptually- human" argument. This argument suggests that our very concept
of person is inextricably linked to our experience of a human life. [FN99] We have never encountered any nonhuman *1260
persons.
One line of reply to the
persons-are-conceptually-human argument is to develop a theory that advances
criteria of personhood that are independent of the criteria for being
human. For example, it might be argued
that the criteria for personhood are possession of second-order beliefs and
possession of second-order desires--beliefs about one's beliefs and desires,
the objects of which are one's own
first-order desires. [FN100]
In the legal context we are imagining, other
lines of reply to the persons- are-conceptually-human objection are
available. First, our inquiry is
focused on legal rather than moral personhood. Although we may lack experience
with moral persons who are not human, we have extensive experience with legal
persons, such as corporations, that are not natural persons. This answer is not satisfactory,
however. The concept of moral
personhood may well be relevant to the question whether AIs should be given
certain constitutional rights; although the legal question is not the same as
the moral one, the two are likely to be interrelated.
Second, and perhaps more importantly, we are
imagining a future form of life quite different from our current
situation. Today, one can only imagine
nonhuman entities that might be persons.
The second scenario imagines a world in which we interact frequently
with AIs that possess many human qualities, but lack any semblance of human
biology. Given this change in form of
life, our concept of a person may change in a way that creates a cleavage
between human and person. Our current
linguistic practice will not be binding in the imagined future. In other words, one cannot, on conceptual
grounds, rule out in advance the possibility that AIs should be given the
rights of constitutional personhood.
The argument against constitutional personhood for AIs also might be developed in the following way: "We are humans. Even if AIs have all the qualities that make us moral persons, we shouldn't allow them the rights of constitutional personhood because it isn't in our interest to do so." [FN101] Call this the "anthropocentric" argument. I do not know quite *1261 what to say to this argument. It seems to reject the idea that we could have moral obligations to anything that is not a human--