Author: CEO & Founder of Trusli, Gloria Qiao

‍We've frequently discussed the benefits and pitfalls of ChatGPT. It's only prudent to remind folks about the risks of "hallucinations" and the occasional lapses in accuracy that ChatGPT can encounter. It's no surprise that some "ingenious" lawyer already attempted to use it to cite case law and was caught on the spot when an astute judge had their staff cross-check the references.

Those skeptical of ChatGPT are now laughing uproariously at this blunder. "I told you so", they say. And yes, it's absurd to ask ChatGPT to cite case law and to trust its references without double-checking the sources. After all, let's remember that ChatGPT is a large language model—it generates content based on context. Precision and accurate sourcing aren't its strong suits. Producing plausible-looking content is more its way of operating.

‍

Does this misstep mean that lawyers should abandon ChatGPT? This idea couldn't be further from the truth. Legal documents are, by nature, constructed on context. So, ChatGPT excels at creating templates, generating different drafts of a particular clause, reviewing documents, and suggesting relevant edits, to name just a few uses.

‍

So, how should lawyers utilize ChatGPT without tripping up like our colleagues at Levidow, Levidow & Oberman? Here are a few suggestions:

1. Pose the right questions.

‍

As we've mentioned before, ChatGPT operates on a “garbage in, garbage out” principle. If you don’t ask the right questions, or if you ask the wrong ones, you can expect disappointing responses. You need to start with the right role, context, and instructions. This can be trickier than it seems in a legal context. 

For example, try asking ChatGPT to "draft me a master supply agreement". Sure, you'll get a master supply agreement, but how good will it be? Where does ChatGPT gather its sources to create it? Is it robust? Does it favor the supplier or the customer?

Instead, consider a more specific prompt like, “Imagine you are my general counsel of a large car manufacturer with lots of leverage over my car part suppliers. I want to create a template that I use with all my suppliers of automotive parts, with robust terms in my favor. In particular, I want my warranty terms to be compliant with the automotive industry standard, so that if anything goes wrong with the supplier-supplied parts, I can go after them with full resources in terms of limitation of liability and indemnification. I want my agreement to be precise, and tightly written without being overly lengthy or containing too many vague provisions without actual legal meaning.” 

Would you get a perfect MSA like one drafted by a seasoned partner from a New York law firm? Probably not. Will you get a decent MSA? Most likely. With this approach, you're likely to get a better starting point than if you tried to create one from scratch. Therefore, we've always believed that proper "prompt engineering" is half the battle.

2. Develop your own model.

‍

Given the importance of asking the right questions, having the right level of knowledge and expertise is vital for prompt engineering. If you’re unsure of what questions to ask, creating a smaller model before turning to prompt engineering may be a viable solution. 

Here's how it works: gather a collection of precedents containing the key terms you want to focus on. With these precedents and some data extraction, you can distill the right questions to ask. In the case above, you can gather 20 agreements containing the type of provisions you care about such as warranty, indemnification and limitation of liability. Combine these provisions with “entities” such as type of supplier, leverage of the supplier, type of commodity being purchased, etc. Then, you can “train” your small model to ask the questions, such as the one we built above. Of course, the actual nuances of how to “extract” the entities, what “entities” to extract, and how to “train” goes beyond the scope of this article. If this approach sounds fascinating, please reach out to us. 

Here at Trusli, we have developed our proprietary approach regarding how to do this, as well as many other sophisticated methodologies, such as simulation-based training. We are excited to help you train your small models and deploy GPT to automate your legal operations. 

3. Continuously refine and verify your work. 

 We can't emphasize enough the importance of checking the content generated by ChatGPT before sending it to another party. Not doing so is not only risky (due to potential inaccuracies) but also unprofessional. Once the content is generated, a human should always verify critical provisions, check sources of information, and even fine-tune the wording. If not, we may face more mishaps like that of our friends at Levidow, Levidow & Oberman. Remember, lawyers are licensed by the bar association, not ChatGPT, because we are ultimately accountable and liable for the content we create.

In conclusion, we're fully aware of the limitations and flaws inherent to ChatGPT. However, to dismiss it entirely due to these shortcomings is not wise. The abilities of large language models far surpass the human capacity for processing vast amounts of text, generating relevant cross-references, and drafting improved and more sophisticated language, among other things. As the saying goes in Chinese, one should not abstain from eating for fear of choking. We shouldn't abandon the potential benefits of ChatGPT due to a few mishaps arising from insufficient skill and professionalism in its implementation. Instead, we should embrace the tide of large language models and forge ahead.