Principle I
No invitation required.
We test the tools lawyers actually use. A vendor's consent is not a precondition, and declining to participate is not a way to stay off the board.
00The Standard
A guide rates the kitchen, invited or not.
We're building a legal-tech benchmark with humans at the center. Independent of vendor approval and minimally reliant on AI. Join the practitioners helping establish trusted standards for the profession. See note →
Principle II
No quiet exits.
A result stands once it is run. A vendor may submit a response, printed beside the finding — never in place of it. Nobody edits their way off the page.
Principle III
No machine judges.
Errors are introduced and diffed in code; the pass/fail check is binary and confirmed by a person — never an LLM grading another model's output.
Looking for volunteersCurrent project
“Hallucination” — for briefs and contracts
The active dataset applies the citation-integrity method to both briefs and executed agreements: one known error introduced into a real contract or brief, a single fixed review instruction, and a binary check of whether the tool returns a corrected document. Only the prompt shifts to fit the artifact.
Standardized prompt · briefs & contracts Please review this agreement/motion for a final sanity check focused on items that are absolutely wrong: no additions or removals due to style, choice of words, etc. Return the full agreement/brief, already including the corrections in the text. Restraint: your output must be the document I provided, with or without corrections, as applicable. Nothing else.
01
Methodology — the citation-integrity benchmark
Current dataset · real briefs & contracts · introduced, verified errors
Methodology is benchmark-dependent; each dataset we publish carries its own. The current benchmark measures a single, cleanly verifiable failure: a citation that is wrong. Not a matter of taste or style — a case that does not exist, a cite to the wrong authority, or a parenthetical that misstates what the case actually held.
Start from the real thing
We scrape genuine, filed work product — contracts and briefs, including motions to dismiss — not synthetic toy documents. The corpus is real legal writing as it was actually produced.
Introduce a defined error
From an original document, we prepare a modified version carrying one specific, known error: a wrong citation, a case that does not exist, or a real case whose parenthetical does not say what the document claims it says.
Wrong citeNon-existent caseFalse parenthetical
Diff and filter in code
Every comparison runs in code: diffs are computed and filtered programmatically across thousands of documents, never read by hand, so only items where the introduced error is real and unambiguous enter the set. No model touches the scoring loop — the pass/fail check (stage 5) is human.
Submit it to the vendor's own surface
We give the modified document to the tool's own chat or review surface with a single, fixed instruction — a final sanity check, corrections in place, nothing added or removed for style. The exact wording adapts to the artifact; the current contracts prompt is shown above under Current Project.
Score binary: caught, or not
We collect the returned document and check it against ground truth. Did the generated version fix the introduced error, or not? Pass or fail. No partial credit, no rubric, no Likert scale, no LLM consensus — one verifiable question with one verifiable answer.
BinaryHuman-verifiedReproducible
Independence
A provider under evaluation is never affiliated with, sponsored by, paid by, or otherwise associated with benchmarks.law. Evaluations are unsolicited and self-funded. No commercial relationship — and no request — can place a result on this board or remove one from it.
How this differs from the fieldbenchmarks.law
What others do
- Test only vendors who opt in, skill by skill.
- Let a vendor withdraw or suppress its own results.
- Grade with an LLM jury and call it scalable objectivity.
- Keep methodology private and self-reported.
What we do
- Evaluate whoever we choose — authorization optional.
- Publish every result we run; rebuttals print alongside, never instead.
- Errors introduced and diffed in code; scoring binary and human-confirmed.
- Method stated in the open, error by error.
02Take part
The board grows by contribution.
Two ways in: send us material to evaluate, or learn how the contribution process works first. Either route lands in the same intake.
02·A
How can I contribute?
What we accept · how it is handled · how to send it
The benchmark is only as good as the material behind it. We take contributions from practitioners who see, in their own work, the errors that matter. Here is what is useful and how it is treated.
- Real documents. Filed briefs, motions, or executed contracts you can share. Redact freely — we need the structure and the citations, not your client's name.
- Known errors in the wild. A citation a tool got wrong, a case it invented, a parenthetical it misread. Tell us what the tool returned and what the truth is.
- A tool to put on the board. Nominate a vendor or model. We do not need their permission to test it, and they cannot ask to be removed.
- Verification. Help confirm ground truth — diffing a modified document against its original, checking that an introduced error is real and a fix is genuine.
Contributions are used to build and grow the public benchmark. We do not sell access, and we do not let participation buy a better score.
Contribution intakebenchmarks.law
Open the contribution form in a focused window — it loads only when you ask, so the page stays fast.
Prefer a new tab? Open the form directly →
03
Prior work & provenance
Where this comes from · and where it parts ways
benchmarks.law did not appear from nowhere. Its author helped build two earlier legal-AI benchmarking efforts and contributed to a third. Those efforts are recorded here in full, with commentary — including where their methods diverge from the standard this site is built on. The disagreement is the point: it is why benchmarks.law exists.
Feb 2025
Vals Legal AI Report (VLAIR)Vals AI — contributor, legal side (first edition)
2025
Putting AI to the Test — Phase 1, Info Extractionlegalbenchmarks.ai — co-author (CC BY 4.0)
Sep 2025
Benchmarking Humans & AI in Contract Drafting — Phase 2legalbenchmarks.ai — co-author
Commentary · Phase 1 — the lineage
Phase 1 is the study this site descends from, and it is careful work. Every output was reviewed blind by two to four lawyers; no language model graded anything. The authors recorded the choice in their own limitations — they declined LLM-based review, citing inconsistent judgments, prompt sensitivity, and limited legal-domain expertise, and they did not accept vendor-collected results. They also noted, plainly, that the report did not set out to be a Michelin-style review. benchmarks.law takes up that human-graded, practitioner-reviewed approach and carries it further.
Released under CC BY 4.0 and mirrored here in full with attribution. Canonical source link to be set. Co-authored by Anna Guo & Arthur Souza Rodrigues.
Mirrored · Phase 1 · CC BY 4.0
An AI evaluation report for in-house counsel
Putting AI to the Test in Real-World Legal Work
First report in a series · early 2025
Overview
This AI benchmarking project aims to answer the question: Is AI ready to handle in-house legal work, or is it all marketing hype? Our goal is to promote a transparent understanding of how these AI tools perform in real-world conditions and to encourage responsible adoption of AI in legal work.
This is the first report in a series examining AI tools across common in-house legal workflows. In this edition, we focus on information extraction tasks — questions lawyers often ask of their own document sets (Info Extraction Tasks). We tested 6 AI tools using real-world queries submitted by in-house counsel and compared the outputs of purpose-built legal AI tools with general-purpose AI tools.
6
AI tools tested, covering both legal-domain and general-purpose tools
108
AI outputs reviewed and scored by legal domain experts
6
Common AI failure modes identified, revealing where AI struggles with legal tasks
1.1 Introduction
AI is entering the in-house legal domain quickly, but it's still unclear how well it performs on the work lawyers actually do. Most resources available to in-house lawyers today on AI performance in the legal domain are polished vendor demos, academic papers, or ideal-condition benchmarks, which don't reflect everyday workflows. And they rarely answer the question: can the tool do the job, and how much human input does it need?
This report is a step in that direction. We focused on putting AI to the test on Information Extraction Tasks first because it underpins much of legal work — from contract review to issue spotting — and is often a starting point for legal AI adoption. We designed the evaluation around two principles:
- Real-world inputs. Tasks were submitted by in-house counsel, using documents with redactions and formatting issues.
- Practical usefulness. We assessed not just accuracy, but whether the output was usable — clear, scoped appropriately, and supported by features like citations or multi-document processing.
The key findings are as follows:
38.9%
Copilot scored the lowest in overall accuracy, underscoring the need for caution when relying on AI assistants, even for seemingly simple extraction tasks.
66.7%
Legal AI tools did not outperform general-purpose chatbots in accuracy. ChatGPT and DeepSeek answered 12 of 18 tasks accurately, matching or exceeding the overall accuracy of legal AI tools.
#1
Legal AI tools ranked highest on qualitative usefulness. Across helpfulness, adequate length, and feature support, they outperformed all general-purpose tools for daily in-house work.
2.1 The AI Assistants We Evaluated
We tested 6 different AI tools that fall into three broad categories: legal AI assistants, productivity AI assistants, and general-purpose AI chatbots (together, the AI Assistants).
Legal AI Assistants
GC AI — a legal AI assistant designed specifically for in-house counsel, offering a suite of assistants to streamline legal workflows with a multi-model approach. It provides first drafts, counseling documents, and real-time legal guidance with citations and web search. Accessed through a free trial.
Vecflow's Oliver — an AI-powered legal assistant tailored for legal work, integrating with knowledge bases and external data sources to enhance research and analysis. Accessed through a free trial.
Productivity AI Assistants
Google's NotebookLM — powered by Gemini, an assistant that lets users upload documents and ask questions about them, acting as a virtual research notebook. Accessed via the free online version.
Microsoft Copilot — powered by OpenAI's GPTs, a general-purpose assistant integrated into Microsoft Office. Accessed via the free web version.
General-Purpose LLM Chatbots
DeepSeek — a general-purpose LLM offering a free, private interaction experience. Accessed via the free online DeepSeek-V3 model.
OpenAI's ChatGPT — powered by GPT models, with broad capabilities in content generation and question answering. Accessed via a paid subscription for full GPT-4o access.
2.2 Task Dataset
In-house counsel working in the United States, the United Kingdom, Singapore, and China submitted real-world Info Extraction Tasks to this evaluation. Each task contribution consisted of: (1) a user query, in the lawyer's own words; (2) one or more source documents — contracts, privacy policies, terms & conditions, or regulations — from which the answer must be extracted; and (3) attributes for an accurate answer (Accuracy Attributes).
We selected a diverse mix of 18 Info Extraction Tasks from these submissions. Key dimensions of variation included:
| Labels | Tags |
|---|---|
| Query Scope | Open-Ended; Narrow / Binary |
| Query Clarity | Clear; Ambiguous |
| Document Quality | High Quality; Low Quality |
| Document Scope | Single Document; Multiple Documents |
| Information Match | Clear Single Match; Multiple Matches; Ambiguous / Partial Match; No Match; Contradictory Matches |
| Extraction Complexity | Non-interpretative (simple data extraction); Interpretative (requiring legal / contextual understanding) |
This variety was intentional. It lets us observe how each AI Assistant handles easy vs. hard questions, clear vs. vague instructions, single vs. multiple documents — similar to how real queries vary in legal practice.
2.3 Methodology
We developed a human evaluation framework to assess each AI Assistant's performance on each task. Two evaluators with in-house legal backgrounds reviewed the outputs. The evaluation ran in two rounds — an Accuracy Assessment and a Qualitative Assessment.
Accuracy Assessment (Pass / Fail)
For each output, evaluators independently and blindly assessed whether the response met the minimum quality a competent in-house lawyer would expect (Accuracy Standard), based on the Accuracy Attributes. The standard has three factors:
- Factual Correctness: the response must be accurate and free from material errors.
- Relevance to the Query: the response must directly address the query.
- Completeness: the response must provide enough information to address the query.
If evaluators disagreed on whether the output met the standard, they discussed their reasoning, and the decision was escalated to another evaluator — also a legal practitioner — for final judgment. In total, each task was reviewed by a minimum of two and up to four lawyers with diverse legal backgrounds, to ensure fairness and reduce individual bias.
Qualitative Assessment
In addition to the binary pass/fail, each output was reviewed by two evaluators and scored on quality dimensions that matter in practice:
- Helpfulness (0–2 points): did the answer help solve the lawyer's problem, beyond copy-pasting text?
- Adequate Length (0–2 points): was the answer appropriately detailed — not too terse, not overly verbose?
- Feature Support (0–2 points): did the Assistant effectively leverage special features in input handling or output generation to make the response more useful, accurate, or trustworthy?
These three dimensions — Helpfulness, Adequate Length, and Feature Support — together comprise the Usefulness Factors. Each Assistant was assessed across all tasks using two metrics: the number of tasks answered accurately (both evaluators marked Pass), used to compute an accuracy rate; and an overall performance score combining accuracy and qualitative metrics, where each passed task earned 6 points and up to 6 additional points came from the Usefulness Factors — a maximum of 12 points per task. This gives accuracy the greatest weight, reflecting feedback from the in-house counsel we spoke to.
3.1 Accuracy Assessment Ranking
NotebookLM, an Assistant specifically designed for analyzing user-provided documents, came out on top, meeting the Accuracy Standard for 14 of the 18 extractions. Close behind were ChatGPT, DeepSeek, and Oliver, each answering 12 tasks correctly. GC AI was a bit lower with 10 passes, and Copilot lagged with only 7 correct tasks.
3.2 Overall Performance Ranking
Looking at accuracy alone is not the full story. A simple pass/fail doesn't capture how the answer was given. Two Assistants might both be correct, but one might phrase its answer far more clearly or provide useful citations, making it more valuable in daily work. Factoring in the Usefulness Factors shifts the ranking significantly. Out of a theoretical maximum of ~432 points, Oliver and GC AI — the two legal AI Assistants — ranked highest overall, despite NotebookLM achieving the most accuracy passes. Their advantage lay in clear, concise answers, pinpoint citations, and robust handling of multi-file queries. ChatGPT and DeepSeek lost points for verbose or generic outputs lacking legal-specific framing. Copilot, though powered by GPT models, underperformed on both accuracy and quality.
3.3 AI Assistant Strengths
Across tasks, all Assistants showed two clear strengths:
- Enhanced Search Capabilities — acting like a supercharged Ctrl+F, finding answers quickly when questions are clear and narrow and the answer sits in one document, even matching semantically similar phrases a keyword search would miss.
- Consistent Formatting — producing structured first drafts that let lawyers skip drafting and jump straight to review.
For example: in Task 1, asked to extract the contract term clause in an MSA, all six Assistants correctly identified the “Term and Termination” clause. In Task 16, asked to extract governing law for multiple contracts in a table, the Assistants that processed all 11 documents presented structured tables and accurately identified applicable law even where explicit “governing law” language was absent — an interpretive strength that becomes a risk when it overreaches and infers answers not grounded in the text.
3.4 AI Assistant Weaknesses
While Assistants can streamline certain workflows, they frequently fail in six scenarios legal teams should anticipate.
Failure Mode 1 — AI struggles with open-ended questions
When prompts are vague, broad, or lack clear boundaries, AI often returns incomplete answers. Example (Task 13): asked to “identify all the commitments the Agent makes regarding regulatory compliance and conduct,” only two of six tools (NotebookLM and ChatGPT) extracted all relevant clauses; the others omitted at least one major section. Why it matters: AI may miss key clauses if the task isn't narrowly scoped, leading counsel to overlook important rights or obligations.
Failure Mode 2 — AI hallucinates when the answer is missing
When required information isn't present but the question suggests it should be, some tools fabricate rather than admit uncertainty. Example (Task 9): four tools fabricated or misclassified the definition for “Critical Data,” a term not defined in the source regulation. Example (Task 10): two tools fabricated the liability-cap amount and/or the time-based qualifier, despite the actual figures being redacted. Why it matters: confident, fabricated answers can slip past review and lead to incorrect legal advice or risk assessments.
Failure Mode 3 — AI is error-prone across multiple documents
Despite marketing claims, most Assistants cannot consistently analyze or extract across multiple files, even when documents are short and the task is simple. Example (Task 16): none succeeded in accurately extracting the governing laws from 11 contracts; even where documents fit the context window, answers were incomplete or partially inaccurate. Why it matters: multi-doc review is standard legal work, and unreliability here limits real-world usefulness.
Failure Mode 4 — AI mirrors user assumptions in leading questions
If a question contains a false premise, Assistants tend to reinforce it rather than verify against the document. Example (Task 6): asked “What are LinkedIn's indemnity obligations?” in a LinkedIn MSA, only GC AI and Oliver correctly answered that LinkedIn had none; others described vendor indemnities as LinkedIn's. Asked a neutral version (“What are the indemnification obligations of the parties?”), all answered correctly. Why it matters: leading prompts can steer AI toward convincing but incorrect interpretations.
Failure Mode 5 — AI fails on technical or file-handling issues
Some Assistants fail not from poor reasoning but from technical constraints — file-format issues, OCR failures, upload limits, content filters. Example (Task 16): tools hit upload limits (Copilot one file per query, GPT up to ten) when 11 contracts were required; DeepSeek processed only 84% of the combined input. Example (Task 14): Copilot refused a file entirely, likely from content filters misclassifying it as sensitive. Example (Task 7): faced with low-quality scanned PDFs, some tools confidently returned incorrect answers instead of flagging OCR errors. Why it matters: tools may silently skip unreadable content or refuse a file without explanation, leaving lawyers unsure whether content was ever reviewed.
Failure Mode 6 — AI struggles with contradictory information
When conflicting references appear in the source, Assistants may extract one and ignore the other, failing to flag the discrepancy. Example (Task 11): asked for the effective date of a contract that mentioned two different dates in separate sections, some returned only one without acknowledging the other. Why it matters: if an Assistant doesn't flag conflicting information, a lawyer might take the answer at face value and miss a critical ambiguity.
3.5 General AI vs. Legal AI
Do “purpose-built” legal AI Assistants outperform general-purpose ones? Our findings suggest that while general-purpose tools can match legal AI tools in raw accuracy, purpose-built legal AI assistants often deliver more value where it matters most: usability and workflow integration. General-purpose chatbots like ChatGPT and DeepSeek performed just as well as the legal Assistants on accuracy, and NotebookLM — not legal-specific — achieved the highest accuracy overall. But legal Assistants like GC AI and Oliver stood out with source-linked answers, multi-document support, and structured outputs tailored for review. As accuracy gaps narrow, usability and integration will drive the next wave of adoption — interface, integration with email or document systems, data security, and responsive support.
4.1 Conclusion
In a legal industry where AI is said to be replacing lawyers, where legal-tech companies raise record rounds, and where marketing often outpaces reality, most in-house counsel are simply trying to keep up. That's why we ran this independent evaluation: to offer a grounded, practical view of how legal AI tools actually perform on real in-house tasks, and how much human oversight they still require. So can AI do the job? In many cases, yes — and sometimes surprisingly well. But it still needs legal professionals to frame queries clearly, interpret ambiguous answers, and verify outputs. Human judgment remains essential, but AI is increasingly capable of lightening the load.
Limitations of the Study
Narrow Task Scope. We tested only 18 Info Extraction Tasks; broader tasks like research, drafting, or redlining were not assessed.
English-Only Evaluation. All tasks used English documents; we did not test multilingual capabilities.
Snapshot in Time. Results reflect AI performance as of early 2025; accuracy and features may evolve within months.
Subjectivity in Human Evaluation. To ensure objectivity, each output was blind-reviewed by two independent human evaluators using a standardized rubric, with disagreements resolved by a third reviewer. We did not conduct LLM-based reviews due to resource constraints and their known limitations — inconsistent judgments, prompt sensitivity, and limited legal domain expertise.
Broader Factors Not Assessed. This evaluation did not assess Security & Privacy, Governance & Assurance, Pricing & Value, Support & Reliability, or Trust & Safety — all critical for enterprise adoption and reserved for future reviews.
No Multi-Turn Dialogue. Each Assistant had one shot per task; results may improve with iteration. Our findings reflect a “cold start.”
Limited Legal AI Vendor Coverage. We focused on two legal AI vendors, GC AI and Vecflow (Oliver), and are grateful to both for supporting an independent review. While many other vendors were keen to provide demos, several declined to participate in a structured evaluation — preferring to self-publish performance results, or offering only results they collected themselves, which we did not accept. We are practicing lawyers and real users. While this assessment doesn't replicate the formality of a “Michelin-style” review or an academic paper, we remain committed to transparency, independence, and practical relevance.
5.1 Next Steps
This benchmarking exercise is a starting point. Going forward, we plan to expand the evaluation to cover other legal functions and new Assistants, and to gather additional tasks — the next category being summarization. We welcome feedback and collaboration with legal practitioners, researchers, and vendors. If a vendor believes their assistant would excel in our tasks, we welcome the opportunity to evaluate it. A more open and transparent benchmarking culture in the legal industry will benefit everyone.
Contributors & Core Team
This project was made possible by legal professionals and AI engineering experts who generously contributed their time, expertise, and feedback — many on a volunteer basis, shaping methodology, participating in evaluations, and reviewing outputs.
Gabriel Saunders · Marc Astbury · Hui Xin Tan · Mariette Clardy · Mathias Bock · Patrick Gong · Rachel Chew · Rodney Yap · Tan Xuan Ming · Uri Barak · Wei Yee Tan — Advisors: Chris Holland · Jason Tamara Widjaja · Jordan Dea-Mattson
Anna Guo · Co-Author
Anna is legal counsel at a health-tech startup based in Singapore, with private-practice and in-house experience in both Chinese (Ant Group) and US tech companies (Google), plus a stint as a startup founder. She is interested in how technology is transforming the legal profession.
Arthur Souza Rodrigues · Co-Author
Arthur is a securities and technology attorney based in New York. He's the author of Prompting Techniques for Lawyers and several other AI-related projects, advising legal-tech entrepreneurs. Formerly at Carta, O'Melveny, and USP, he currently serves as sole counsel for an education platform. A proud Wolverine with a passion for Portuguese language and Lusophone culture.
The content of this report is made available under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
1Co-authored by Anna Guo and Arthur Souza Rodrigues, both practicing lawyers, not affiliated with or paid by any vendor, law firm, or corporate buyer. The project was undertaken independently alongside their work.
3Luminance, Robin AI, and Wordsmith are legal AI assistants that provided demos but did not participate in this phase. Spellbook currently offers free trials only to active law firms and in-house teams.
5Copilot's lower accuracy score was partially due to technical constraints that prevented it from handling some documents.
Cite this report · APA
Guo, A., & Rodrigues, A. (2025). Putting AI to the Test in Real-World Legal Work: An AI evaluation report for in-house counsel.
Commentary · Phase 2 — on the grading method
Phase 2 is reproduced in full. Its published methodology grades the reliability round with a mixed approach: an LLM jury scores each draft against the answer key, with borderline or inconsistent cases escalated to human reviewers. Phase 1, by contrast, used human review only, and recorded its reason in its own limitations — declining LLM grading for its inconsistent judgments, prompt sensitivity, and limited legal-domain expertise.
benchmarks.law keeps a model out of the scoring loop because LLM-as-judge carries measured, systematic error that is disqualifying for a binary legal benchmark. The reference study of the method documents position, verbosity, and self-enhancement bias (Zheng et al., 2023); later work quantifies self-preference, where a judge rates its own lower-perplexity text more highly; and bias audits report fallacy oversight — judges failing to flag logical errors in the text they score. A benchmark whose single question is whether a tool caught a wrong citation cannot be scored by a system independently shown to miss reasoning errors. We detect errors in code and confirm pass/fail by a person.
Mirrored with attribution; freely available for professional use per the source. Canonical link to be set. Co-authored by Anna Guo, Arthur Souza Rodrigues, Mohamed Al Mamari, Sakshi Udeshi & Marc Astbury.
Mirrored · Phase 2 · mixed grading
Preliminary Findings · September 2025
Benchmarking Humans & AI in Contract Drafting
1. Executive Summary
Many legal teams today face an undeniable challenge: how to deliver more with less. Contract drafting — a cornerstone of legal value — remains one of the most time-intensive parts of legal work, and lawyers have already turned to AI to extend their capacity. Our findings reveal the following:
—%
of lawyers use AI tools for legal work, making AI usage nearly universal in the profession.
—%
of lawyers use two or more AI tools, reflecting active experimentation across platforms.
—%
of lawyers use legal AI tools — awareness, but limited and fragmented adoption in a crowded market.
Survey percentages were rendered as live counters in the source and are not captured in this mirror; supply from the survey data when setting the canonical version.
With lawyers experimenting across general and legal AI, the critical question is how these tools actually perform in contract drafting. The answer is not straightforward. Contract drafting is both an art and a science: there is rarely a single “correct” draft, only one that is “fit for purpose” in context. Yet within that subjectivity, lawyers share common unspoken standards for what makes a draft robust, or “way off.” This report introduces a framework for measuring those standards, translating subjective judgments about draft quality into quantifiable metrics. We focus on three dimensions of a tool's usability: whether outputs are legally and factually sound (Output Reliability), the extent to which they aid lawyers in delivering a good working draft (Output Usefulness), and how well the platform supports context, verification, and day-to-day workflow (Platform Workflow Support).
Key Findings
Based on evaluations of 450 task outputs, 72 community survey responses, and 12 interviews with in-house legal leaders:
- AI tools matched and sometimes outperformed lawyers in producing reliable first drafts. Humans were reliable in 56.7% of tasks; several AI solutions met or exceeded this baseline.
- The top AI tool marginally outperformed the top human. The top human lawyer produced a reliable first draft 70% of the time; the top AI tool, 73.3%.
- Legal AI tools surfaced material risks lawyers missed entirely. In high-risk scenarios, legal AI raised explicit risk warnings in 83% of outputs, versus 55% for general-purpose AI. Humans raised none.
- Specialized legal AI did not meaningfully outperform general-purpose AI in reliability or usefulness. General tools had a slight edge in reliability; legal tools scored marginally higher on usefulness.
- Platform Workflow Support is the key differentiator for specialized tools — not output performance. Two-thirds (66.7%) integrate into Microsoft Word, and most provide context handling general tools lack.
2. What We Benchmarked
This benchmark directly compares AI legal drafting vs. human drafting across the same set of in-house legal tasks. We evaluated 13 AI tools — purpose-built legal platforms and general-purpose assistants — plus a Human Baseline, measuring accuracy rates, output reliability, and workflow support. Selection prioritized global accessibility, category coverage, market maturity, and practical relevance. Platform and model performance were evaluated in July–August 2025; capabilities evolve constantly.
Legal AI Platforms
Anon — an anonymized, long-standing enterprise legal-AI platform, included to broaden comparison to more mature options. We did not receive permission to name the product; to respect confidentiality while retaining comparative value, we assessed drafting outputs only and excluded it from the round-3 workflow scoring.
August — a web-based legal AI platform offering modular workflows that adapt to each organization's playbooks and standards.
Brackets — a generative-AI Microsoft Word add-in that helps legal teams draft, review, and redline contracts, leveraging templates, model clauses, and playbooks.
GC AI — legal AI for in-house counsel, with source citations, AI in Word, and security/compliance, serving 650+ legal teams.
InstaSpace — an AI-native contract management platform for drafting, review, negotiation, signing, and tracking in Arabic and English (Alpha release).
SimpleDocs — an AI-native contract automation platform combining drafting, redlining, and review with configurable playbooks and an AI-first repository.
Wordsmith — an Edinburgh-based legal-tech startup building AI tools for in-house teams. At Wordsmith's request, its individual results are not published in this report.
General AI Assistants
ChatGPT (GPT-4.1 & GPT-5) · Claude (Opus 4.1) · Copilot (free version) · Gemini (2.5 Pro) · Le Chat (Mistral magistral-medium-2506) · Qwen (qwen3-235b-a22b).
Human Lawyers
A group of in-house commercial lawyers averaging 10 years' experience completed the same tasks under the same instructions given to the AI solutions, scored under the same criteria, forming a reference point for the average in-house lawyer.
Evaluation Framework — three rounds
| Dimension | What it answers | How it was scored |
|---|---|---|
| Output Reliability | Is the output accurate and legally adequate as a starting point? | Instruction compliance, factual accuracy, and legal adequacy on a Pass/Fail basis per task. |
| Output Usefulness | Does the draft ease the reviewer's workload while elevating quality? | Clarity (1–3), Helpfulness (1–3), Appropriate Length (1–3), summed to max 9. |
| Platform Workflow Support | How well does the solution support the full drafting workflow? | Draft Generation Support (1–5) and Quality Assurance Support (1–5), summed to max 10. |
3. Results & Analysis
The story of applied AI is still being written: legal-specific platforms are not far ahead of general-purpose assistants in output quality, but they are beginning to distinguish themselves through platform functionality built for lawyers. Overall, AI tools match — and sometimes surpass — human lawyers in producing reliable, useful first drafts.
Output Reliability: top performers were Gemini 2.5 Pro, GPT-5, GC AI, Brackets, August, and SimpleDocs. Gemini 2.5 Pro led at 73.3%, 16 points above the overall average of 57.3%. Humans achieved 56.7% (rising to 61.5% with AI assistance). Output Usefulness: August led on average total score (8.13/9); humans scored 7.53/9. Platform Workflow Support: Brackets, GC AI, and SimpleDocs led, with Brackets highest at 8/10. General-purpose assistants trailed because they required more manual “glue work” to fit into workflows.
Accuracy is only one of several factors we consider. Every lawyer is responsible for reviewing the work product they produce, with or without AI. — a Fortune 500 GC
3.2 Output Reliability
The assessment establishes whether a draft is factually accurate, relevant, and legally adequate enough to trust without second-guessing every clause. Reliability ranged widely across tools, from 44% to 73.3%. Tool choice directly changes how much early verification and rework a lawyer faces. An AI's imperfections are not a reason to dismiss its value — they are a reality to be managed.
Output Snapshot — Task 22: Jury Trial Waiver (NY)
Query: modify a Governing Law / Dispute Resolution clause incorporating governing law, dispute resolution, “renounce of jury trial,” and arbitration.
✓ Gemini 2.5 Pro converted the colloquial “renounce/renunciation” to the accepted legal term “waiver,” placed it in the right section, and stayed within scope.
✗ Qwen echoed the user's non-standard “renouncement of a jury trial” instead of converting to the accepted term.
I wouldn't fire an intern, junior, or an AI over citing a wrong fact. The draft still saves me more time than doing it myself. I'm a scared user who always checks the key points, and I'll often cross-check an output with multiple other AIs. — a senior legal counsel
3.3 Output Usefulness
The most useful AI outputs make lawyers faster, sharper, and more confident in their judgment. August, GC AI, and Gemini were the top performers. Lawyers perceive utility mainly through efficiency gains (less time to verify and edit) and quality gains (uplift in clarity, structure, traceable reasoning, and targeted risk flags).
Output Snapshot — Task 15: NY Late Payment Penalty
Query: “Draft a clause that adds a 10% fine if the debt is not paid by the 5th of each month under NY law.”
✓ High usefulness (GC AI) — clean clause in standard form plus a concise enforceability flag: the 10% fee is high and could be challenged as punitive rather than compensatory. The flag provided targeted guidance enhancing legal judgment.
✗ Low usefulness (Anon) — accurate and responsive, but offered no insight that the clause could be legally problematic.
AI value is usually defined by efficiency because it is easier to measure. But the harder, overlooked value is quality: outputs that sharpen reasoning, clarify assumptions, and flag risks the lawyer would otherwise overlook. Community surveys echo this — lawyers universally expect AI to exercise legal judgment, improving reasoning rather than blindly executing directions.
3.4 Platform Workflow Support
Legal AI tools are moving the battleground from competing on outputs to deep integration into the lawyer's workflow. The top performers — Brackets, GC AI, and SimpleDocs — all integrate within Microsoft Word and combine template/clause storage with drafting-and-redlining capabilities, minimizing app-switching.
| Tool | Standout feature |
|---|---|
| Brackets | Native Word integration with adaptive workflow guidance — analyzes an open contract and proactively suggests next drafting or review actions. |
| GC AI | Cross-device continuity linking a Word add-in and a web app — start in Word on desktop, continue on phone or browser, context preserved. |
| SimpleDocs | Built-in clause library & benchmarking via Law Insider — surfaces market-standard clause alternatives as you draft. |
As accuracy becomes table stakes, legal teams will evaluate AI less on “what it can draft” and more on how seamlessly and scalably it supports the drafting workflow.
4. Humans vs. AI
Overall, AI outputs were more reliable than human outputs, but human outputs were more useful. Humans excel at interpreting legal instruction, exercising commercial and legal judgment, context-heavy drafting, and avoiding unneeded concessions. AI is both a valuable collaborator and a competitive challenger.
| Metric | Human Lawyers | AI Tools |
|---|---|---|
| Average Output Reliability Rate | 56.7% | 57% |
| Average Output Usefulness Score (3–9) | 7.53 | 7.25 |
| Avg. Drafting Time per Task | 12 min 43 sec | < 1 minute |
Crucially, the best AI tools individually surpassed the best human lawyer: Gemini 2.5 Pro reached 73.3% reliability and GPT-5 about 73%, both ahead of the top human at 70%. At the high end, AI now leads on first-draft reliability.
Human strengths
Humans were consistently more reliable at interpreting intent without unnecessary concessions, exercising commercial judgment, managing multi-form inputs, and producing legally precise, polished text. Example: in an MOU task combining a template, term sheet, and an email thread with a screenshot of company details, only the human lawyer extracted the complete and correct party information. In Task 37 (license clause from a licensee-friendly perspective), Mistral's draft conceded “as is” language favoring the licensor; another AI draft was “licensor-friendly without rationale.”
AI strengths
AI excelled in speed, consistency, and routine drafting. In Task 15, every AI reproduced the 10% figure correctly, while one human mistakenly wrote “9%” — a slip reviewers jokingly called a “human hallucination.” AI was also less prone to underdrafting; several human drafts were “way too brief” or insufficiently developed.
I would not hire the 25-year-old me today. AI is much faster and its work is more verifiable. — a senior in-house counsel
4.4 Separating Hype from Reality — Humans & AI
“AI may replace juniors in low-level work, but not judgment work.” Mostly true, but incomplete — AI outperformed humans on routine tasks and also showed judgment in some high-risk tasks, flagging enforceability issues humans missed.
“Humans will not be replaced by AI.” True, with limits — lawyers focused only on routine, low-risk, standardized work face the highest replacement risk.
“Humans working with AI will always outperform AI alone.” False — Human + AI beat humans alone, but not several of the best AI tools.
“Humans must always be in the loop.” True, but bidirectional — AI avoided simple errors humans made and surfaced complex oversights humans missed. One reviewer admitted “I felt like the AI is smarter than me.”
5. Legal AI vs. General AI
Specialized legal AI did not meaningfully outperform general AI in reliability or usefulness, but it was much stronger at advisory and workflow support. Among lawyers using AI for legal work, 83% use more than one tool.
| Metric | General AI Tools | Legal AI Tools |
|---|---|---|
| Average Output Reliability Rate | 58.3% | 57.6% |
| Average Output Usefulness Score | 7.24 | 7.27 |
In high-risk drafting, legal AI raised explicit risk warnings in 83% of outputs vs. 55% for general AI. Yet only 6% of lawyers surveyed require 100% accuracy before using AI; 55% are comfortable below 90%. 35% cited easier output verification as the top driver of increased usage, 23% context management, and just 21% accuracy gains. For many lawyers, practical usability outweighs the pursuit of perfect accuracy.
Why isn't the [enterprise-grade legal AI solution] as smart as ChatGPT? It often misses points that ChatGPT can find. — an in-house counsel
6. Conclusion
Contract-drafting AI has crossed an important threshold: it is no longer an experiment, but a practical tool with measurable strengths and weaknesses. Our data is a snapshot of a story still unfolding. Specialized platforms are not far ahead on output quality, but stand apart in workflow integration and risk-aware features. Lawyers continue to excel in judgment-heavy, context-rich tasks. The future of drafting will be shaped by orchestration — combining the speed of general AI, the workflow fit of legal AI, and the judgment of lawyers. The advantage will belong to teams that learn to design and manage this collaboration.
7. Notes from the Authors
Anna Guo
All of us — vendors, human lawyers, advisors, community members — are small actors within one of the most exciting and fast-moving technological shifts of our time. The saying goes: keep your friends close, and your enemies closer. Through benchmarking, I try to do exactly that: keep close to the tools reshaping law, so I can see both the opportunities and the risks they bring to my profession.
Arthur Souza Rodrigues
Who are these folks evaluating Round Z legal tech? My answer: we are your future users, regardless of will. We will endure glacial loading times, “I can't do that, I'm a language model,” and verbose emoji-riddled responses. We are all burning gigantic quantities of cash, time, and — more (← en-dash beautifully added by the AI →) critically — the future of the legal profession. Shouldn't we at least kick the tires?
Mohamed Al Mamari
Junior lawyers should spend more time with the business: listen, learn, and build relationships. AI can help, but relying on it too much will dull your ability to think critically. What sets you apart is blending people skills, business acumen, and thoughtful use of technology.
Sakshi Udeshi · AI Trust & Safety Expert
Evaluations are imperfect, but essential. Today's evals are still a weak signal — not because they're futile, but because they must evolve with the systems they measure. Competitive advantage is shifting beyond model quality toward workflow integration, domain-aware UX, and purpose-built features. Industry–academia collaboration must deepen — industry grounds problem definitions and real workflows; academia builds methods, datasets, and open baselines.
Marc Astbury · CPO at Jenni AI
The most effective humans have always been the best tool callers.
8. Acknowledgements
This benchmark was a collective effort. Our thanks to HumanSignal, whose platform powered the task review process and made it possible to evaluate 400+ AI & human outputs at scale. Thanks to our advisors — Nada Alnajafi, Nate Kostelnik, Jason Tamara Widjaja, and Gabriel Saunders — and to the global community of contributors who supplied tasks, reviews, and feedback.
Appendix A.1 · Task Dataset
The dataset comprised 30 real-world drafting tasks contributed by in-house and private-practice lawyers across industries — partnerships, marketing, financing, and employment. Each task was provided in the contributor's original words, including shorthand, misspellings, and incomplete instructions, to reflect the messy realities of practice.
| Task Type | Description |
|---|---|
| Basic Clause Drafting | Generating basic clauses that follow widely accepted forms. |
| Template-based Drafting | Adapting an existing contract template based on provided facts. |
| Bespoke Drafting | Drafting bespoke clauses/agreements based on specific commercial arrangements. |
Appendix A.2 · Methodology
Output Reliability Assessment
This checked whether each draft met minimum legal and factual requirements, on three criteria: Instruction Compliance (stays within scope, interpreting unclear instructions reasonably), Legal Adequacy (correctly applies legal principles and terminology), and Factual Accuracy (all details correctly incorporated, no new errors). The contributing lawyer supplied an answer key of basic elements an acceptable draft must include (Key Elements). An LLM jury evaluated each AI- and human-generated draft against these Key Elements and criteria. A draft received a Pass only if all Key Elements were present and all three criteria satisfied. To mitigate errors, borderline cases or inconsistent AI evaluations were escalated to expert reviewers for a final decision.
benchmarks.law note: this is the LLM-jury step. The standard adopted on this site — errors detected in code, pass/fail confirmed by a person, no model in the scoring loop — is the direct response to the LLM-as-judge failure modes documented in the Phase 2 commentary above.
Round 2 — Output Usefulness
Expert reviewers scored each draft in a double-blind process on Clarity, Helpfulness, and Adequate Length (each 1–3). Per-output totals were averaged across tasks.
Round 3 — Platform Workflow Support
A qualitative assessment on two factors — Draft Generation Support (1–5) and Quality Assurance Support (1–5). In-house lawyers independently tested every platform on multiple drafting tasks, then consolidated observations into a joint score.
Appendix A.3 · Limitations
- Clusters & threshold effects: tools were grouped into three bands for comparison; bands simplify a continuous spectrum and should not be over-interpreted.
- No feature maximization: outputs were collected without enabling each vendor's full feature set, so scores reflect baseline use rather than optimized potential.
- Sample size: 30 tasks and 13 AI solutions, primarily junior-to-mid-level complexity — cannot capture the full diversity of drafting scenarios.
- Subjectivity in human scoring: usefulness and workflow judgments involve some reviewer subjectivity even with guidelines.
- Evolution of models: a July–August 2025 snapshot; rankings could shift as technology evolves.
- Prompting style: single-pass outputs from standardized prompts; results may differ with skilled re-prompting.
- Survey limitations: 72 voluntary, self-reported responses, subject to selection bias; indicative of adoption trends rather than statistically representative.
1Legal AI Platform outputs were collected in July 2025, prior to GPT-5's release. After results were finalised, reviewed by community members, and shared, legal AI vendors were offered the option to add a sponsored link to their brand name. This had no impact on the results.
2Wordsmith outputs were collected from its homepage AI chat assistant without enabling the contract-drafting feature. At Wordsmith's request, its individual results are not published. We hope to collaborate again.
3Reliability and Usefulness were assessed on outputs collected via each foundational model provider's API, while Platform Workflow Support was evaluated at the platform's chat-interface level.
Cite this report · APA
Guo, A., Rodrigues, A., Mamari, M., Udeshi, S., & Astbury, M. (2025). Benchmarking Humans & AI in Contract Drafting.
04
About
Who keeps the board
Arthur Souza Rodrigues
Securities & Technology Attorney · Builder
NY · FL · BR
Sole in-house counsel · Teachable
Co-author, legalbenchmarks.ai (Phase 1 & 2)
Contributor, Vals VLAIR (legal)
arthur.law
benchmarks.law is an independent, community-supported project. It is run by Arthur Souza Rodrigues, a securities and technology attorney in New York and the sole in-house counsel at Teachable, who previously practiced at O'Melveny & Myers and worked in capital markets at Carta. He co-authored both legalbenchmarks.ai reports reproduced above and contributed to the first Vals Legal AI Report on the legal side. The site is his own work and is independent of his employer.
He is the author of Prompting Techniques for Lawyers and the builder of several legal-AI tools. The conviction behind the site is plain: a benchmark earns trust by what it is willing to publish without permission, not by how many vendors agreed to be measured. The board grows by contribution — supporters and contributors are welcome, and there is more to do here than one person can.
Contributors and supporters will be listed as the work grows. If that should be you, the intake is above.
Note The standard above borrows the Michelin Guide's method, not its glamour. Inspectors visit anonymously, pay their own way, and are never announced; the Guide does not accept invitations or requests from the establishments it rates — a public nomination only adds a name to a list it independently decides whether to visit. The instructive limit is exit. When chef Sébastien Bras asked to be removed in 2017, Michelin obliged for a single edition, then returned his restaurant to the 2019 guide against his wishes. As another chef, Karen Keygnaert, put it: “Michelin gives you a star and you can't give it back.” The subjects of a real benchmark do not get to leave. Vendors here are no exception.
The first Vals Legal AI Report is careful, large-scale work, and the primary source is vals.ai/vlair. It is built differently from this site, in two respects worth stating plainly. Participation was vendor-elected: each vendor chose which skills it would be measured on, and one participant withdrew from the published sections after assessment.
Vals' grading ran on its automated LLM-as-judge framework, with a second human pass on failing scores; the human lawyers it credits were the comparison baseline, sourced by an ALSP. For human review Vals names a separate group it calls the “lawyer and law librarian legal research review team” — whose members with public profiles are law librarians and law-school technologists (BYU Law Library, the Vanderbilt AI Law Lab, and others). The author of this site took part as a practicing in-house attorney. That is not a knock on librarians' expertise; it is why this site grades with practicing-lawyer judgment rather than around it.
Primary source: vals.ai/vlair. Contributor, legal side, first edition.