Legal AI gets only 54% right: is that enough?

In that test, Astra for Law passed the overall correctness check on 54 percent of the questions. GPT-6 Astra with web search alone reached 38.7 percent. The difference is substantial: roughly fifteen percentage points. It suggests that a general-purpose model connected to the internet is not enough; source selection, instructions, and domain context make a significant difference to the result. Yes, but let’s not jump from there to a comforting conclusion. An overall correctness rate of 54 percent also means that, on that specific benchmark, almost half of the answers failed that check. It is a clear improvement, not a license to stop reviewing. That is probably the most honest figure in the entire presentation. The tool seems useful precisely because it acknowledges, at least indirectly, that legal research still needs expert oversight.

Full episode: https://youtu.be/Ja0Q4F-tsxs

🤖 AI-generated content: the script, voices, and images in this episode were produced using artificial intelligence tools.

#Shorts