Negotiating the use of Technology Assisted Review/Artificial Intelligence in document review

By Tracy Ickes

In litigation today, it is commonplace to deal with hundreds of thousands—sometimes even millions—of electronic files and documents. Reviewing these materials for production is a long and expensive process. “Technology assisted review” (TAR) has the potential to streamline review, but parties should carefully negotiate protocol for its use.


Several companies offer “technology assisted review” (TAR). Documents are uploaded to a database in which reviewers code each document for responsiveness, privilege, or other customizable tags. TAR uses AI to analyze how documents are coded and, based on that, it predicts the likelihood that a given document will be responsive or not. TAR can also predict the documents that are related to one another according to any number of criteria. Some TARs operate based on a “seed set” of coded documents, while others continue to monitor and learn from coding in real time.

Parties can utilize TAR to prioritize documents for review, or even to set thresholds below which there will be no manual review. For example, if TAR predicts that only 1% of the documents in a set of 10,000 documents are likely to be responsive, a party may review a sample to validate the TAR’s prediction, and then determine whether it is worth the time and expense to review the rest. TAR has the potential to save significant expense by eliminating the need for manual review of documents that are unlikely to be relevant.

Important issues for consideration

How do you know the “smoking gun” is not mixed among the set of documents TAR suggests have a low likelihood of responsiveness? If you find out your opposing counsel was not reviewing documents based on a recommendation from TAR, do you suspect he or she was just looking for a reason not to inquire into something potentially problematic? A recent decision from the District Court of New Jersey indicates that parties should disclose the use of TAR and agree on parameters for its use at the earliest opportunity. In re Valsartan, Losartan, and Irbesartan Prods. Liability Lit., 2020 WL 7054284, – F.R.D. – (D.N.J. Dec. 2, 2020) (“Valsartan”).

In Valsartan, the parties entered into a stipulated “Electronic Discovery Protocol” that required them to meet and confer “as early as possible” to discuss search terms for electronic documents as well as “any TAR/predictive coding prior to using any such technology to narrow the pool of collected documents to a set to undergo review for possible production.” After a year of negotiating search terms, and just weeks before the first rolling production was due, a defendant (Teva) informed the plaintiffs that it would be using a “continuous multi-modal learning platform,” which would monitor coding of documents and prioritize documents for review. Teva also stated that it would inform plaintiffs if it decided not to review a population of documents that were unlikely to be responsive. The plaintiffs objected that they never would have agreed to limited search terms if Teva intended to use TAR.

The court began by noting that TAR is an “appropriate discovery tool” and agreeing with the “general notion that a party can use TAR so long as its use is transparent and timely disclosed, and the parties collaborate in good faith about its use.” While the court recognized Teva’s argument that setting aside TAR would require a disproportionate document review effort, the court concluded that the parties’ agreed-upon Electronic Discovery Protocol trumps the standard proportionality analysis.

The court ultimately limited Teva’s use of predictive coding because it failed to meet and confer about it “as early as possible,” as required by the parties’ protocol. But in considering what remedy to fashion, the court declined to order manual review of all documents. (“To put it bluntly, the thought that Teva or plaintiffs might have to spend millions of dollars to manually review irrelevant or marginally relevant documents is more than mildly disturbing.”) Instead, the court found a middle ground: Teva had to comply with a protocol for predictive coding its experts helped negotiate in the summer (a protocol that Teva ultimately rejected), and Teva had to produce a subset of documents marked non-responsive so the opposing party could validate the predicative coding on its own.

Key Takeaways

There are a number of lessons in the Valsartan decision. Counsel should determine early on whether a case might benefit from TAR and negotiate a very clear protocol for use of TAR with opposing counsel. Factors to consider include:

  • The type of TAR/AI that will be used;
  • Whether TAR will be combined with, or be exclusive of, search terms;
  • Whether and with whom sample sets may be shared to validate TAR;
  • At what point the parties agree that further manual review is unnecessary;
  • Whether any disputes will be governed by a proportionality analysis; and
  • Any agreed-upon remedies for violation.

As the decision in Valsartan shows, courts may be reluctant to order manual review even if TAR is used in violation of (or without any) governing protocol. Thus, in considering remedies to request, parties should consider what search or validation techniques could resolve the dispute, as well as standard remedies for discovery violations.

We continue to follow developments in the use of TAR and AI in litigation.

author img


Tracy Ickes


Posts By this author