We are often asked if we incorporate artificial intelligence (“AI”) into our legal workflows and electronic discovery processes. This question is not surprising given the efficiencies and cost savings associated with AI.
Typically, these questions are followed by inquiries into how the AI tools work and their defensibility. That is, how the use of AI can be defended if its use is challenged by a judge or opposing party. Essentially, the defensibility of AI tools and their corresponding results boils down to the ability to explain the results in plain language.
As a starting point, we first need to establish what is meant when we say AI. Oftentimes, it is generically described without any meaningful distinction between available tools. AI is nuanced with several tools that do not necessarily relate to or depend on one another; therefore, we must understand the selected tools before we even get to defensibility.
To better understand, let’s break down some commonly used AI tools in legal applications:
While not exhaustive, the above list illustrates how varied the uses of AI can be. Given this, there is no one-size-fits-all for the defensibility of AI. However, the AI workflow outlined below sheds light on how we can make AI explainable and in turn defensible.
First, all decisions, processes, or procedures undertaken to use AI need to be documented. It is difficult to explain how AI tools work if the steps taken to achieve results are not readily apparent. Oftentimes, AI tools are proprietary so accounting for all decisions within your control is especially important. For example, we are not able to modify the algorithms that create models; however, we can determine the reasonable number of documents needed for training a new model.
Documentation also leads to repeatability, which can help validate results achieved from AI tools. For instance, let’s say that our use of CAL is questioned by opposing counsel. Opposing counsel is concerned that our processes may have missed a substantial number of responsive documents. Of course, we could point to the statistical significance of the sampling conducted on the unreviewed universe to show it contains a reasonable percentage of missed responsive documents. However, we could also conduct a new sample using the same parameters to bolster our conclusion that the reasonable percentage will not significantly vary if different documents are reviewed.
Next, consider the sources of data that the AI tools are analyzing. Are we missing a key source of data that could throw off the analytics? To illustrate, let’s return to our example from anomaly detection. If the employee used a secondary email address that was not analyzed, then the results could have identified a behavior that only existed because we did not review the entirety of the employee’s email data. Have we properly cleansed the data before AI tools are used? Data cleaning means issues in the dataset are fixed or removed to prevent negative outcomes with the AI tools. Data cleaning may include removing duplicates or fixing formatting errors with metadata.
Related to data considerations, we should look for any potential bias with the artificial intelligence used. One source of bias is derived from our data selection. If incomplete, the analysis may not be representative of the entire dataset we are trying to apply our AI tool. Bias related to the algorithms behind the AI tools may also exist. Consider how models can be influenced by the decisions of human reviewers. Bias could be introduced when a model is trained using incorrect coding decisions of human reviewers. To prevent this, subject matter experts should be used to train models.
After using one of the above AI tools, we must validate that it generated the desired or expected result. When we say desired or expected result, we mean that the results are reasonable and reliable given the AI tool and its configuration. Validation varies greatly depending on the AI tool selected.
For example, validation of CAL is driven by metrics. Validating that CAL has reasonably identified all responsive documents requires elusion testing. An elusion test is a random sample of the portion of the document universe that won’t be reviewed by a human or produced. Elusion testing is used to show that the percentage of responsive documents in the unreviewed universe, or the rate of eluded documents, is reasonable and that further efforts are not necessary to satisfy a party’s discovery obligations.
We can compare the rate of eluded documents to other metrics, such as richness, to show that using CAL was a reliable method for finding responsive documents. We may need to go back to the documents depending on our analysis of these metrics.
On the other hand, some AI tools may be validated using expertise, such as AI translation. In that case, someone who is fluent may need to validate AI translation to confirm that the translated text accurately reflects the meaning of the document. Otherwise, the AI translation may not be reliable, or defensible, for human reviewers to use the translated text to code documents.
Finally, we need to succinctly describe the above processes if the issue of defensibility ever arises. Remember, the inability to show defensible practices for AI tools could result in added expense if the court requires you to go back into the documents.
Must Lawyers Supervise the Robots? (The Legal Ethics of AI)
Statistical Sampling in Legal Document and Data Reviews