MODEL TRAINER: JDS: Training the next generation of specialist legal LLMs

The first debate about synthetic data for large language models was largely about scale.

Could one powerful model generate enough instructions, examples and answers to help train another model?

The answer increasingly appears to be yes—but with an important qualification. The value of synthetic data depends much less on whether it was generated by AI than on how it was designed, filtered, validated and combined with other data.

That matters for legal AI because law has a particularly difficult training-data problem.

Legal organisations hold enormous volumes of valuable material, but much of it is confidential, privileged, commercially sensitive or affected by privacy and professional obligations. Public judgments and legislation are important, but they do not reproduce the working file a legal model encounters in practice: correspondence, witness accounts, contracts, forms, financial records, expert reports, incomplete instructions and documents that contradict one another.

Justice Data Studio is designed to help fill that gap by producing rich synthetic legal matter files that can be used not only to test legal AI, but also to train and fine-tune specialist models.

What the research has shown

Several influential projects established the modern case for synthetic LLM training data.

Self-Instruct showed that a model could generate instructions, inputs and outputs, filter invalid or repetitive examples, and use the resulting dataset for instruction tuning. The researchers reported a 33 percentage-point improvement over the original GPT-3 model on Super-NaturalInstructions, demonstrating that carefully filtered model-generated examples could substantially improve instruction following.[1]

Stanford Alpaca made the idea concrete and reproducible. The project fine-tuned a 7-billion-parameter LLaMA model on 52,000 synthetic instruction demonstrations generated in the Self-Instruct style. Its preliminary evaluation found behaviours similar to a much larger proprietary instruction-following model, while also emphasising unresolved safety, licensing and quality limitations.[2]

Microsoft’s Textbooks Are All You Need work pushed the argument further: data quality and curriculum design can sometimes matter more than raw scale. Phi-1 was trained using selected web data together with one billion tokens of synthetically generated “textbook-quality” explanations and exercises, achieving strong coding-benchmark performance despite being much smaller than contemporary frontier models.[3]

The emerging legal literature is beginning to apply the same logic to domain adaptation. SynLexLM, for example, combines synthetic legal question-and-answer generation with curriculum learning and parameter-efficient fine-tuning. Its early work illustrates both the opportunity and the unresolved challenge: synthetic data can target scarce legal capabilities, but legal accuracy, bias, generator quality and rigorous held-out evaluation remain critical.[4]

There is also a major warning in the literature.

Research published in Nature found that recursively training models on generated data can produce model collapse: rare information disappears, errors compound and the model gradually learns a distorted version of the underlying distribution. Preserving original data reduced degradation in the researchers’ experiments.[5]

The lesson is not that synthetic data should be avoided. It is that synthetic training data must be curated as an engineered dataset—not treated as an unlimited stream of automatically generated text.

Why legal training data needs to look like a matter, not a prompt

Much synthetic instruction data follows a simple pattern:

question → answer

That is useful for teaching a model to respond to instructions. But many valuable legal tasks involve something much richer:

matter file → factual reconstruction → source selection → legal workflow → reviewable output

A legal model may need to determine which document contains a relevant fact, distinguish an allegation from evidence, identify a missing attachment, preserve chronology, recognise inconsistent accounts, avoid privileged material and generate an output that remains traceable to its sources.

Those capabilities cannot be trained adequately from isolated legal questions alone.

Justice Data Studio can generate coherent synthetic files containing combinations of:

• pleadings, applications and response documents;

• correspondence and email threads;

• contracts, policies and business records;

• witness statements and interview notes;

• expert, medical or financial material;

• spreadsheets, schedules and structured metadata;

• incomplete, irrelevant or noisy documents;

• deliberate contradictions and edge cases; and

• known ground truth, chronologies, evidence indexes and model answers.

The critical feature is the relationship between the documents.

A useful synthetic matter is not simply a folder of plausible-looking files. Dates must align. People and entities must remain consistent. Attachments must exist. Later documents must reflect what earlier participants knew. Intentional contradictions must be distinguishable from generation mistakes. The ground-truth record must identify what actually happened and what cannot properly be inferred from the available evidence.

That creates training material for much more than legal Q&A.

How rich synthetic files can support specialist legal models

A single synthetic matter can be transformed into multiple supervised training tasks:

• extracting source-linked facts from documents;

• building chronologies and evidence indexes;

• classifying documents and procedural stages;

• identifying contradictions, gaps and unresolved questions;

• separating known facts, allegations and inferences;

• generating structured intake or triage outputs;

• producing forms, letters, summaries and working drafts;

• learning when to use retrieval or another tool;

• learning when information is insufficient; and

• learning when a task must be escalated for human review.

The same matter can also support preference or correction data. A model can be shown a grounded output and a superficially fluent but defective alternative—for example, one that invents a fact, overlooks a contradiction or relies on a privileged document—and be trained to distinguish between them.

For agentic legal systems, the file can become an environment rather than a static dataset. A model can be trained to inspect the document index, retrieve a source, ask for missing information, populate a structured record and stop when the evidence does not support the requested conclusion.

The importance of local and specialist deployments

Over the next one to three years, legal organisations are likely to deploy more specialist models inside controlled environments.

Some will be fine-tuned open models. Others will use parameter-efficient adapters such as LoRA, private retrieval systems or smaller models trained for a narrow task. They may run in a firm-controlled cloud tenancy, a sovereign environment or, for sufficiently compact models, on local infrastructure.

The drivers are straightforward: confidentiality, data sovereignty, predictable cost, lower latency, control over model changes and the ability to optimise a model for a particular jurisdiction or workflow.

General-purpose frontier models will remain important. But a smaller model trained to perform one legal process reliably may be more useful than a much larger model that knows more in general but has never seen the organisation’s task structure.

The bottleneck will often be data.

A firm may have thousands of historical matters but be unable to use them freely for training. Even where use is legally permissible, the material may be inconsistently organised, poorly labelled or too sensitive to send through an experimental pipeline. Public legal corpora rarely include the complete operational matter record needed to teach workflow behaviour.

Synthetic matter files provide another option: build the training curriculum deliberately, with known facts, controlled variation and no real client information.

What responsible use should look like

Synthetic data is not automatically accurate, unbiased or privacy-safe merely because it is synthetic.

A defensible training programme should therefore:

• preserve clear provenance for every generated matter and training example;

• use human QA and domain review;

• separate training, validation and genuinely held-out test families;

• prevent near-duplicate scenarios from leaking across those sets;

• maintain real or independently curated evaluation data where lawful and appropriate;

• test for bias, missing tail cases and over-represented patterns;

• document the generator models, prompts, filters and dataset versions;

• retain known-defect and intentional-contradiction registers; and

• measure whether fine-tuning improves the target task without degrading other important behaviours.

Justice Data Studio’s role is therefore not simply to generate more text.

It is to create curated legal learning environments: realistic files, controlled facts, answer keys, failure cases and training tasks that can be used to teach specialist systems how legal work actually unfolds.

As local and domain-specific legal models proliferate, that infrastructure may become as important as the models themselves.