What the Good Place Teaches Us About Good Legal Design
It’s the quirky show on Netflix blending moral philosophy with flying shrimp and Kristen Bell. Sounds like a recipe for disaster, right? Wrong. It’s surprisingly good and heading into its fourth season.
For the uninitiated, The Good Place is premised on the idea that regardless of faith, there is a ‘good place’ or a ‘bad place’ people go to when they die depending on how they lived their life. Our protagonists (Eleanor, Chidi, Tahani and Jason) have ended up in the ‘Bad Place’ and are now on a quest to get into the Good Place which leads them to the pivotal question: how was their fate decided?
***Spoilers ahead (Season 3)***
In season 3, our heroes discover the decision was made by ‘Accountants’, ethereal beings much like real world accountants who measure the total sum of a person’s actions while alive. Points are obtained for good deeds and deducted for bad ones. Sounds simple enough. Yet there is one important detail: no one has been accepted into the Good Place in over 521 years. Holy mother forker.
Turns out, the Accountants have been applying the same algorithm since the dawn of time and there is now something deeply wrong with the outdated system. The simple act of buying a shirt could result in loss of points because the purchase inadvertently supports slave labour.
Allusions to social credit systems aside, this scenario highlights some important issues in legal design of automated systems in law.
1. Always Review the Underlying System
Technology won’t fix a system - but it sure can scale one. Most people assume digital transformation is just automating existing business processes. But if we don’t review the underlying system carefully we can take tiny errors that are easily corrected in a manual system and scale them to the magnitude of something that can affect the integrity of the organisation. Centrelink’s Robo-debt crises is a good example of this.
It is also possible that an issue that didn’t exist on deployment may arise some time later as occurred in The Good Place. Legal designers should build in failsafes such as regular algorithm review dates to avoid the unintended and potentially devastating consequences of scaling mistakes in automated processes.
2. Adopt Human-Centred Design
The Accountants in The Good Place viewed their human subjects as nothing more than units to be measured. They never cared enough to consider whether the original algorithm was still worth applying and quite happily condemned all humans to eternal damnation for over 500 years. Yet nothing could be more important to our heroes.
If human centered design isn’t adopted in the creation of external facing systems, then the likelihood of scaling errors is high. Simply speaking to users about how they will be affected is not enough as users themselves often can’t anticipate how the system will impact them. It’s not enough to simply consult. Designers should adopt techniques such as Walk-a-Mile and persona mapping to really anticipate issues that may arise.
3. Transparency
If your system is making a determination then transparency is key. As determinations usually depend on data, privacy is a pivotal design constraint. People are going to want to know what is being collected and why. This isn’t just a GDPR issue. It’s a UX issue. Just like our foursome in The Good Place, users want to know how a system decides. Simply knowing the ethereal Accountants determination wasn’t enough, they had to dig deeper into the detail to understand how judgment was delivered in order to decide whether or not to accept the result.
Research has consistently shown that parties are more accepting of the results of disputes where they felt the process was fair. Fairness is discerned in part by transparency about the rules and expectations of the process. It seems likely this same principle applies to determinations made by artificial intelligence. It stands to reason that if you’re automating decision making then transparency is required. At Inkling, we pride ourselves on helping firms adopt these principles by rethinking their processes, incorporating human-centred automation, and prioritising their scalability.
Is your firm overdue for a redesign?
First published 12 March 2019