Maximising AI Recall in Legal Tech

The legal technology sector is rapidly evolving with the integration of advanced AI tools like GPT-4. Greg Kamradt's recent study sheds light on how GPT-4's recall performance varies with token count and document positioning, offering valuable insights for legal professionals looking to leverage this technology.

Key Findings from the Study

1. Recall Performance Degradation: Kamradt's study revealed that GPT-4's recall starts to diminish significantly when handling documents over 73K tokens, roughly equivalent to 110 pages of text. This finding is crucial for managing lengthy legal documents.
2. Importance of Document Positioning: The study highlighted that GPT-4 recalls information placed at the beginning of documents more effectively. However, the recall rate drops for facts in the middle sections (7%-50% of the document).
3. No Guarantees on Recall: The unpredictability in specific fact retrieval underlines the necessity for critical human oversight.
4. Less is More for Context: Providing GPT-4 with reduced context can enhance accuracy, an essential consideration for drafting or reviewing legal documents.
5. Fact Placement Matters: The positioning of crucial facts within documents significantly impacts GPT-4’s recall efficiency.

Applying These Insights to Legal Tech

1. Document Management: When preparing legal documents for AI analysis, it's advisable to structure them with the most critical information at the beginning. This approach can optimise the AI’s performance in tasks like summarisation or issue identification. Long term this could signal a change in how contracts are structured in order to support AI systems (and also people) attempting to make sense of them.
2. Review and Analysis: In the short term legal professionals should be mindful of the limits when using AI for reviewing lengthy documents like contracts or reports. Breaking down larger documents into smaller sections can ensure more reliable AI performance. However long term this will be a problem for vendors/in house development teams to solve
3. Drafting Assistance: For AI-assisted drafting, placing key points and summaries at the start of a document can help in maintaining accuracy and relevance.
4. Data Processing: When processing large volumes of legal data, it’s beneficial to segment the data effectively. This helps in avoiding the information overload that can hamper GPT-4’s recall capabilities.
5. Training and Development: Legal teams should train their staff on these insights to use AI tools more effectively. Understanding the limitations and strengths of AI in document processing can significantly enhance efficiency and accuracy.

Summary

Greg Kamradt's study provides critical insights into the operational limits and efficiencies of GPT-4, particularly relevant for the legal tech sector. By understanding and adapting to these nuances, legal professionals can greatly enhance their use of AI, leading to improved accuracy, efficiency, and overall performance in legal document handling and analysis. As AI continues to evolve, staying abreast of these developments is not just beneficial but essential for the modern legal practice.