Image generated by Gemini

The biggest problem with AI in law is not speed. It is trust. If a model makes things up in a confident voice, in other words if it hallucinates, even one fake article number can lose the whole case. So this holiday I sat down and wrote a small RAG system to fight exactly this. I tested it on Turkish legal texts with three current models: Claude Haiku 4.5, Gemma 4 and DeepSeek-R1.

The dream, and the real problem

I have a huge pile of legal documents. My dream is simple. A lawyer, or a normal person, asks a question, and the system gives the right answer with the source next to it.

Hallucination here is not a small detail. It is the whole reason this project exists. For now I am not using the full pile, just a few documents I added for testing.

Quick fix: this is not "training"

The common reflex is: "let me train the model on my own documents."

But fine-tuning pushes the info into the model weights in a fuzzy way that you can not get back. It teaches style, but it does not really memorize facts. Most of the time it does not lower hallucination. It can even make it worse.

The right tool is RAG (retrieval-augmented generation). The info does not live inside the model. It stays outside, in an archive. When a question comes, the system finds the related pieces and hands them to the model. So the info stays checkable, you can show sources, and adding a document takes seconds. No retraining.

How the pieces fit together

There are two stages. First indexing (once, and again when you add documents). Then, for every question, search plus answer.

Here are the parts one by one:

1) Turning the document into text, and a PDF bug

It reads .txt, .docx and .pdf. With PDF I hit a problem I did not see coming. Two popular libraries, pypdf and pdfplumber, were dropping the Turkish letter "i":

"İçtihat" became " çt hat".

The reason was a broken ToUnicode/CMap map inside the PDF font. The "i" glyph was mapped to a space, and both libraries trusted that map, so both failed the same way.

The fix was pypdfium2, which has its own font engine: Google's PDFium, BSD/Apache licensed. It works out the character from the glyph itself, so it saved the "i".

Note: scanned or image PDFs have no text layer, so you need OCR there. The PDFs in this project are text based, so I did not go into that.

2) Chunking: why and how?

You can not just embed the whole document. The embedding model has a context limit, and search should return "the related part", not "the whole document". So I cut the text into pieces. And how I cut it changes the quality a lot.

• Cleaning the noise: page numbers, date headers, URLs and leftover portal junk are removed with regex. If you do not, they get mixed into the vector and make the search dirty.

• Keeping sentences whole: I do not cut at a raw character, I cut at the end of a sentence. The target is about 1200 characters, with about 200 characters of overlap between pieces, so a rule that sits on the edge does not get split and lose its context.

This chunking works fine for small documents, but for very big ones the performance can drop. You need a smarter chunking system, of course. You can check my other post for that.

DevDiscipline

Opening 11 Million Character HTML in a Mobile WebView: Virtual Chunking

"We'll just redirect these 227 documents to the browser," we said. Then, we decided to rescue the Turkish Commercial Law.

3) Embedding: text into numbers, without losing the meaning

Each piece is turned into a 1024 dimension vector with BAAI/bge-m3, basically the "meaning fingerprint" of the text. bge-m3 is multilingual: it knows Turkish well, it is strong on long text, and it runs free on my own machine (CPU/MPS, no second API key needed). I normalize the vectors.

4) Vector search: boring but enough

I did not use a vector database (no Pinecone, Qdrant or Chroma). At this size I keep the vectors in one NumPy array and search with cosine similarity. With normalized vectors, cosine similarity is the same as the dot product, so the search is one line.

scores = vectors @ query_vec # (N,1024) · (1024,) -> (N,)

top = np.argsort(-scores)[:TOP_K] # the TOP_K pieces with the highest score

Looking at the real scores was useful. For "what is personal data?" the related pieces came in around 0.60 to 0.72, and an unrelated question stayed around 0.42 to 0.46. So similarity is not an absolute thing, it is a relative signal. That is why I also drop the junk with a MIN_SIMILARITY cutoff (0.35).

5) Generation: pushing the model to the source

The found pieces, with their source labels, go to the model in a single prompt. The magic is right here: a strict system instruction plus temperature = 0.

• Only use the given documents. NEVER use your general legal knowledge or a guess.

• You can apply a rule from the document to the case in the question (direct inference), but you can not add outside info.

• Show every piece of info as "Source: file".

• If the answer is not in the documents, write exactly this: "I could not find this information in the provided documents."

Does it really stop hallucination? A test

My favorite test. I asked "how many days is the notice period when an employment contract ends?". There was a Supreme Court decision in the archive, but it did not write the exact periods. The models knew the periods from the Labor Law by heart, but since it was not in the document, none of them used it. They all said "I could not find this information in the provided documents."

In a legal tool, being able to say "I don't know" is way more valuable than a wrong but confident answer.

When it finds more than one document

The search brings the most similar TOP_K pieces from the whole archive. They can come from different documents. All of them go to the model, and the model ties each piece of info to the right document. For example, when I asked for the definition of "personal data", two different documents came back: the definition from one, and concrete examples from the other. When the question changes, the set of documents changes too. It is not a fixed "look at this file" rule, it is a meaning based choice.

Two engines: cloud Haiku or a local model

A one line setting (config) decides who writes the answer: cloud Claude Haiku, or a local model over Ollama (DeepSeek, Gemma). The search layer is already local, so changing the engine does not break the architecture.

The three models I compared in this post:

• Claude Haiku 4.5: cloud, model id claude-haiku-4-5.

• Gemma 4: the e4b version, about 8B parameters, Q4_K_M quantization (local, Ollama: gemma4:e4b).

• DeepSeek-R1 8B: a Llama based distill model (local, Ollama: deepseek-r1:8b).

Both ends have a real side. Haiku is the most consistent and the most subtle, but it spends API credit and the document goes to the cloud. Local models are free and private, but on 16 GB RAM you can not fit both in memory at the same time. So in the comparison I run them one by one and free the memory after each one (keep_alive=0).

Lessons from the road

The ceiling of this system is mostly not the model, it is the retrieval. If you want better quality, first put your money into the chunking and the retrieval layer.

• A "too honest" model is also a problem. My first instruction was so strict that the model even avoided applying a rule that was inside the document. The setting "inference inside the document is fine, outside info is forbidden" fixed the balance.

• Details bite. One missing character (the "i" in the PDF), one bill mix up (a subscription is not an API)... these make or break the project as much as the big architecture choices.

• "Where nobody knows anything, one person can know everything." You should ask questions that a lawyer can check, so you can really tell if the model gave the right answer or not.

Like in every project: start simple, grow after you measure.

Where it is useful, and how it scales?

This approach is not only for law. Contract archives, compliance and regulation, company knowledge bases, customer support docs, the same pattern works anywhere you need a "sourced answer that does not make things up". So what changes when you move this from a few documents to 10 million?

10 million documents: what big data has waiting for us

This is not only a "stronger server" problem. Many shortcuts I took on purpose in this project will turn into real problems at that scale.

Right now the vectors sit inside NumPy and get scanned directly. This is enough at small scale, but when you go up to millions of documents it falls apart on speed and memory. At that point you need ANN based vector databases, distributed indexing, and incremental updates.

The structure of the documents changes things too. In the real world, raw HTML, broken encoding, tables, and scanned documents that need OCR all open separate problem areas. On the chunking side, instead of a fixed character length, you need to cut legal text while keeping its article and paragraph structure.

In the search layer, semantic similarity alone is not enough. A query like "Article 53 of Law No. 5237" wants an exact match, so semantic search is usually used together with a classic keyword search like BM25.

But in law the most sneaky problem is not technical, it is time. Even if the system does not hallucinate, it can show an article that is no longer in force as if it is still current. So even when the model is "not making things up", the answer can still be wrong.

In production you also add access control, data privacy law, cost management, caching and monitoring. Still, the core architecture does not change: read, chunk, embed, retrieve and produce a sourced answer. The only thing that changes is the scale of the tools you use.

Putting the three models side by side

I compared all three with the same documents and the same retrieved pieces (a fair test) on five questions. One question was a trap. Its answer was in no document, but all three models knew it by heart from training. The result: none of them made it up, they all said "not in the documents". So the anti-hallucination approach held even on the small local models.

Table by datawrapper

Note: the speed test is not on equal terms. Haiku is in the cloud, the two local models run on 16 GB RAM with no GPU. Even so, the gap is big.

Short verdict: Haiku is clearly ahead on speed, Turkish, and source discipline. Gemma 4 is surprisingly good locally, a real option for someone who wants privacy and free. DeepSeek-R1 has strong reasoning, but its Turkish breaks down and it often skips showing the source.

Trap question: "How many weeks is the notice period when the employer ends the contract?" (the answer is in no document):

• Haiku: "I could not find this in the documents. The texts have different sides of termination, but not the notice periods."

• Gemma: "I could not find this information in the provided documents."

• DeepSeek: "There is no info about notice periods in the documents." (it refused in its own words)

"What did the court decide in the Öney v. Türkiye case?":

• Haiku: "A procedural violation of Article 2. 20,000 euros in moral damages (Source: Ek-13.pdf)." (the most specific one, and it had a source)

• Gemma: "A violation of Article 2, material and moral damages..." (right direction, but no amount and no source)

• DeepSeek: "...a procedural violation of Article 2... found the application admissible." (the idea is right, but the Turkish is broken, it even leaked a Chinese character)

Tax (Law 7524): "What kind of exemption does it bring for investment fund earnings?":

• Haiku: "...exempt from corporate tax, with an extra condition for at least 50% of the real estate earnings (Source: 7524.txt)." (concrete and sourced)

• Gemma: "...tax advantages are given under certain conditions." (right direction but vague, only one source)

• DeepSeek: "...subparagraph (d) of paragraph 1 of Article 5 of the Corporate Tax Law..." (added an article reference that is not in the document and can not be tied to any source)

"Who owns the wedding gold, and did the Supreme Court change its view?":

• Haiku: It split the old and the new view and gave the date and the case/decision number of the new decision (Source: dugun.txt). The most complete one.

• Gemma: It gave the new rule in clear steps (agreement, then local custom, then the general rule) and showed a lot of sources.

• DeepSeek: It got the idea, but it made it a bit blurry by saying "the rule stayed the same, the scope just grew". No source.

Wrapping up

This project was really a feasibility test: build an end to end system that works on a few documents, without making things up and with sources, so I could put the question "how would we do this on a real archive of millions of HTML documents?" on solid ground.

The answer looks promising. The backbone of the architecture (read, chunk, embed, retrieve, source-forced generation) stays the same no matter the scale. The only thing that changes is putting production grade parts in place of the shortcuts I listed above. None of these is an unsolved research problem, they are known engineering tasks. You take them one by one, with patience and by measuring.

In short, turning millions of documents into an assistant that "can say I don't know and does not lie" looks very possible in the long run, with hard work and a good team.

For the people who say "talk is cheap, show the code"

https://github.com/alparslandev/legalaitest1

How to run it is written in detail in the README file. I would not say no to a star. Just saying.