Old words, new worlds: Notes from a trip to England
Earlier this June, I attended the Institute for Law and AI’s Summer School on Law-Following AI hosted in Cambridge, England. After the conference, I stuck around in London and Oxford for a few days while working remotely. Across all of it, from a summer school on AI and the law to a debate on artificial personhood, I found a pattern of old words and institutions straining to fit new technologies.
Law-Following AI may not be as feasible or desirable as it sounds
There has been a surge of interest in the intersection of law and AI and numerous applications to LawAI’s Summer School and the following Workshop, a testament to the organization’s great work on the subject and the legal field’s reckoning with the myriad questions AI poses. I’m not sure what made my application stand out, but I’m incredibly grateful to have been chosen to attend the Summer School in Cambridge. A few weeks later, I am still buzzing from the conversations and research directions the conference sparked.
Over the course of two days, LawAI’s Director of Research Cullen O’Keefe moderated and led discussions about Law-Following AI (LFAI) with around fifteen law students and early career professionals. LFAI are AI agents that are “designed to refuse to take illegal actions in the service of their principals.”1 You may have heard of alignment, the idea that AIs should be “aligned” with the goals and values of humans. Think of LFAI as alignment specifically with the legal system. An LFAI should be trained to understand the law, follow it, and not be used to circumvent it. During the summer school, we considered the benefits and objections to this approach and its political and technical feasibility.
One issue that piqued my interest is the “task splitting” objection. Consider a bad actor trying to use a law-following AI to commit a crime. The LFAI would be trained not to commit a crime on behalf of the bad actor and not to aid or abet the bad actor in committing a crime. But suppose the bad actor split the crime into ten tasks, each of which is not a crime in itself. Then, they assign ten different LFAIs to handle each of the tasks, without any of them knowing the instructions of another. In this case, the LFAIs would be helping the crime occur even after legal alignment training.
Additionally, there is the issue of fine-tuning and ablation. If a model has open weights, meaning you can download it and run it locally, you can (relatively cheaply and trivially) remove the safety guardrails that the model was trained to follow. Frontier models will likely remain closed for this reason and others, but open weight models are only a few months behind the frontier and have significant capabilities, capabilities which are only set to improve. Even if open-source models were given legal alignment training, they could quickly be reversed and used to commit crimes by a bad actor.
While these two concerns with the feasibility of LFAI came to the fore during the discussion, upon reflection after the conference I considered an objection to the desirability of LFAI. We want AIs to follow laws so that our society is more just, but what if the laws we ask them to follow are unjust? Would a perfectly law-following model have counseled Rosa Parks to not move to the back of the bus as it would be breaking the law? And if LFAI in the United States or other liberal democracies makes sense to us, would we want the AIs of China or North Korea to be so strictly law-following?
No matter how AI agent liability shakes out, it is probably in a frontier model company’s interest to align their models with the legal system to avoid prosecution. I think as a baseline rule, LFAI is a great idea to prevent a wide range of harms that could be perpetrated by or with the aid of AI models. But there are real hurdles to the feasibility of and serious edge cases questioning the desirability of LFAI that we will have to consider as more and more decisions are made by or guided by such systems.
What is personhood, anyway?
While visiting a friend at Oxford, I attended the Oxford Union’s debate on the resolution: Can AI attain personhood?
There was debate on all four of the resolution’s words: “Can” (in the sense of trying to answer the descriptive question rather than the normative question of should), “AI” (some participants claimed that in an email they were informed that the debate was specifically about generative AI, in case that somehow limits the discussion to the current paradigm of AI. My gut tells me that future paradigms will also be generative), “attain” (a lone speaker focused on this and pushed the audience to consider how personhood is something humanity bestows), and “personhood” (a word that, to me, is still ambiguous—some implied personhood required being rights bearing, others that only duties were enough). Some participants focused on legal personhood, others on moral personhood, some on whether personhood is dependent on a biological substrate, others on whether personhood should be bestowed (if it is such a thing that is bestowed).
I am well aware that debates can quickly turn into arguments about definitions, but I was hoping for more substantive conflict rather than arguments whizzing past one another on different planes.
I particularly enjoyed Jeff Sebo’s arguments and oratory and Bret Weinstein’s paradigm-shifting considerations. I met Jeff at the Eleos Conference on AI Consciousness and Welfare in Berkeley last November (which I wrote about here) and was not surprised that he was compelling. I only know Bret through his (sometimes controversial) X threads and expected him to be more divisive, but I found his argument about the inevitability of future, more biological AI paradigms forming and the multi-step process we should have to ensure alignment to be compelling as well.
However, I was dismayed by the overall structure and quality of the debate. There were no responses to previous speakers among the invited participants, and they mostly talked past one another. Among the Union members, only a two-minute response stood out for the strength of its arguments and its oratory. Among the invited debate participants, many read directly from printed speeches, and many arguments did not address the question of can but rather the normative question of whether AIs should have personhood rather than the descriptive one that was given.2 I was hoping for more out of the debating union with such a storied history, an organization which inspired my friends and me when building our own debate club in college.
Perhaps it’s a sign of similar debates to come. The words we have in our linguistic toolkit that we reach to when trying to delineate what AI is or is not are formed through reference to distinctly human-shaped examples (intelligence, consciousness, and personhood, etc). A new kind of thing may render those words useless unless they are stretched, and the question underlying this debate was whether we stretch a word or we leave it to its current range and come up with something else to describe this new thing.
Corporate legal personhood is a great example of stretching a term for pragmatic reasons. The corporate fiction allows humans to continue taking risks and innovating, with companies going under rather than humans going to jail or bankrupt if the company’s products cause harm or the company is unable to pay a debt. In egregious circumstances, you can still “pierce the veil” and go after the human beings behind the corporate fiction. Some debaters seemed to think that as soon as you provide some analogous legal personhood to AI the humans behind the company which created it instantly go scot-free if the AI causes harm, but following the precedent of corporate legal personhood the veil could still be pierced if, say, someone negligently or purposefully neglected safety training for the model.
One thing I did enjoy about the debate was the traditional format of voting. Listeners vote with their feet as they leave the hall through one of two doors, one with “noes” written above it and another with “ayes” written above it. On the other side, union members manually tally the votes as people walk through. As I walked out, I wondered whether this process could be automated by a computer vision model.
In case you were wondering, I walked through the “aye” side. Due to the ambiguity of what was actually being argued, I erred on the side of caution: “Can” provides a mighty wide berth.
Status hierarchies, status hierarchies everywhere!
The Union has recently been embroiled in controversies (yes, those are five different links to five different recent controversies), ranging from election fraud to statements made by leadership that sparked outrage. I am reminded of one of Plato’s critiques of democracy, the Ship of State, in which those who take control of the ship (an allegory for successful politicians in a democracy) are successful because of their rhetorical skills and ability to persuade voters, not because they are any good at actually steering the ship.
The Oxford Union is a multi-million-dollar organization with a beautiful historic campus and all the pomp and circumstance you can ask for. Leadership is elected every term, which lasts just eight weeks. From what I gathered, the organization functions as pageantry peppered over cutthroat elections which stamp in the next generation of would-be Prime Ministers. The never-ending election season and constant cycling through leadership leave precious little time for learning how to run and actually running an internationally renowned debate club. (Any similarity this may seem to have with the current state of U.K. politics is, of course, coincidental).
Oxford might be the Platonic ideal of college
I hope the previous discussions did not give the impression that I did not enjoy my time in Oxford. On the contrary, I really enjoyed exploring colleges and the events my friend Ryan, a visiting student there, brought me to during my brief visit. And the weather complied, thankfully, perhaps giving me a skewed perspective of the average gloominess in England.
I recently read Privilege: The Making of an Adolescent Elite at St. Paul’s School by Shamus Khan, which explores how the elite are educated through the lens of a year at one of the most prestigious boarding schools in America. While I disagreed with some of Khan’s takeaways, I really liked his idea of ease.
Expecting only entitled kids, he found that the vast majority of the elite boarding school students were perfectly nice to be around, and in fact the snooty, arrogant ones were outcasts for having those traits.
This was by design. Khan found that a modern elite education eschews entitlement in favor of ease, a comfortable and relaxed posture towards any situation. The world of the elite is no longer cordoned off from other social strata as it once was, especially in the American mixing bowl of culture. Likewise, eliteness is not signaled anymore by preferring a night at the opera over a night in a nightclub, it’s by going to both and being equally comfortable in each situation. Students come from a surprisingly wide range of backgrounds, but each of them learns the etiquette of a formal dinner without rejecting their background. They learn to see the world as meritocratic, and that they can do anything in it.
I found a version of this at Oxford. Khan’s elite signal ease precisely by taking nothing too seriously; the posture is studied lightness. At Oxford, the history and the formal dinners and the prestige are still taken seriously, and so the students take themselves seriously, too. Taking yourself seriously, in turn, bends the world in your image and provides the sense of ease.
So which is it: do we stretch the old words and institutions until they fit, the way “person” once stretched to cover a corporation, or do we admit defeat and coin new ones? I’ll take the lawyerly way out and say: it depends. The words we have were built around us. The things arriving now are not us, and not quite not-us either. For the moment we’re doing what people have always done when the world outruns the dictionary: arguing, with old words in old rooms, about what the words were ever supposed to mean.
Appendix:
Not having a [Holmesian] bad man as president is probably a good plan”
- LFAI Summer School participant
https://law-ai.org/law-following-ai/, second paragraph of the abstract.
One can argue that this is the right question to ask as personhood may be something bestowed by humans collectively - precious little time was spent fleshing this out.