CLAUSE-CRAFT
A fine-tuned 7B model that extracts 41 contract-clause categories into validated JSON — with a full cost/accuracy analysis.
- Status
- Benchmarked on real hardware
- Role
- Solo — data pipeline, training, evaluation, serving benchmarks
- Timeline
- 2026
At a glance
A fine-tuned 7B language model that extracts all 41 CUAD clause categories from commercial contracts into structured, schema-validated JSON. The project is really a rigorous comparison of fine-tuning versus API approaches, benchmarked on a single RTX 4000 Ada with no synthetic numbers.
- Fine-tunes a 7B base with LoRA, then aligns output format and precision with SFT followed by DPO.
- Benchmarked against regex, a 14B frontier LLM (zero- and few-shot), and a DeBERTa extractive model.
- vLLM serving benchmarks with quantization comparisons for latency and throughput.
- A cost-breakeven analysis showing exactly when fine-tuning pays off versus paying per API call.
- Ships a model card, governance documentation, contamination checks, and a general-capability evaluation.
Key numbers
41
Clause types
Full CUAD taxonomy
Validated JSON
Output
Schema-checked, not free text
5
Approaches compared
Regex, zero-shot, few-shot, extractive, fine-tuned
Real GPU
Reported on
No synthetic benchmark numbers
Tech stack
- Training
- 7B open base modelLoRA adaptersSupervised fine-tuningDirect Preference Optimisation
- Baselines
- Regex heuristicsZero-shot & few-shot frontier LLM (14B)DeBERTa extractive
- Serving
- vLLMQuantization comparisonsLatency / throughput benchmarks
- Hardware
- Single RTX 4000 Ada (20 GB)End-to-end reproducibility scripts
The problem
Contract review is a structured-extraction task: given a document, find every clause of 41 types and return it as data. The open question for a team is whether to fine-tune a small model or just call a big one — and that decision is usually made on vibes.
CLAUSE-CRAFT answers it with measurements.
Training
A 7B open base model is fine-tuned with LoRA adapters. Supervised fine-tuning teaches the task and the output schema; Direct Preference Optimisation then sharpens format adherence and extraction precision on the cases where SFT alone is ambiguous.
Output is validated JSON against a fixed schema — the model is not trusted to be well-formed, it is checked.
The comparison
Five approaches are benchmarked head to head: regex heuristics, a 14B frontier LLM zero-shot, the same model few-shot, a DeBERTa extractive model, and the fine-tuned 7B. Every number is measured on a single RTX 4000 Ada — no synthetic data is used for reporting.
- vLLM serving benchmarks with quantization variants for latency and throughput.
- Contamination checking and a general-capability evaluation to confirm fine-tuning did not damage the base model.
- A cost-breakeven analysis: at what volume does fine-tuning beat per-call API pricing.
What I took from it
Fine-tuning versus API is an economics question with a real answer for any given workload. The engineering value is in producing that answer credibly — reproducibly, on real hardware, with governance attached.
Want the details?
The full source, READMEs, evaluation harnesses, and results are on GitHub.
github.com/dariyaDeepanshu/CLAUSE-CRAFT