Industry Collaboration around Benchmarking/Tracking Gen AI Accuracy etc

As the legal tech world gets to grips with generative AI, building trust and mechanisms that promote transparency are going to be critical for success.

The publication of the recent Stanford report: “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools and the ensuing debate by interview, article and press release has been really interesting.  It can’t be true that the same tool is both 90% accurate and 42% accurate!

This is a fantastic opportunity for us to come together across the legal industry and discuss the issues and – more importantly – to agree a way forward.

Litig (www.litig.org) has launched a cross-industry initiative to explore and hopefully build some solutions that help the whole legal industry.

With the help of Richard Tromans (Artificial Lawyer) and Caroline Hill (Legal Technology Insider), our initial call led to 180 organisations registering their interest and we are now in the discovery phase of the initiative, with in-person and hybrid events planned to bring people together to explore and design possible solutions.  During this phase we will gather views from people in the legal industry with knowledge and experience of generative AI: law firms, suppliers, regulators and representative bodies, academics and in-house teams.  The initial in-person session is planned for 1 August in London.

We will publish a summary of the sessions and of course any output here.