The AI problem your firm hasn't hit yet

If you've tried AI on a real file, you were probably impressed. A paralegal uploads a client's placement file and account statement, asks for a demand letter, and gets a draft that's surprisingly good. That first impression is accurate. On one file, today's AI really is that good.

The problem shows up later, and at scale. Firms that go further start running everything through one consumer chat tool like ChatGPT. Staff save client instructions in its memory, upload more files, and use it for eviction notices, meet-and-confer letters and settlement correspondence. (Putting client files into a consumer account is its own problem. More on that below.)

Once it's working across dozens of matters, the drafts start to drift:

• One client's account number turns up in another client's demand letter.
• A draft settlement letter uses the percentage a different client authorized.
• A default judgment request applies another county's attorney-fee schedule.
• An instruction written for one landlord's three-day notice shows up on a tenancy that needed a 60-day notice.

Staff start re-checking every line, and the time savings are gone. The AI didn't get worse. What changed is how much the firm is feeding it.

Why does AI slip once it's used across the whole firm?

An AI model doesn't learn from your firm's use. Vendors release new versions, but the model answering you isn't studying your files between conversations. When it seems to know something new, like a client's name or this morning's news, that's because the information was handed to it in the moment.

That handed-over material is called context: the documents, notes, instructions and history the AI reads before it answers. Context is the only part the firm controls, and it's the part most tools manage worst.

Why does AI mix up client files?

Chat tools remember by piling things up. Every instruction, upload and correction goes into one growing pile, and the AI reads from it every time someone asks for something.

The research is consistent. Language models are most likely to miss details that sit in the middle of a long input,¹ and they get less reliable as they're given more to read, even on simple tasks.²

A law office's pile is full of look-alikes: dozens of clients with similar claims, matters at the same stage, templates that differ by one paragraph, and client instructions that contradict each other on purpose. The more alike the material, the easier it is for the AI to grab the wrong piece. Those studies measure long inputs, not law offices, but mixed-up client files are what that weakness looks like in one.

Will a newer AI model fix it?

It's tempting to wait for the next release. But today's models already draft well from a complete, correct file. What they don't have is your firm's knowledge: how a file moves from intake to the attorney's desk to filing, what each client has authorized, and which steps your attorneys insist on before anything goes out.

An AI model knows how an average firm works. It knows how yours works only from what you give it, and it can use that only if the right piece reaches it at the right moment. A newer model pointed at the same pile makes the same mix-ups.

What about confidentiality?

How do you keep AI from mixing up matters?

The answer is older than the internet: put things where they belong. When a firm's knowledge lives in organized folders, each with a short written note saying what's inside and how to use it, the AI reads only what the current task needs. A demand letter for one client is given that client's file and that client's instructions, not the rest of the firm's matters.

A 2026 research paper gave this approach a name and a method. I explain it in plain English in the next post.

Common questions

Why does ChatGPT mix up my clients' details?

Chat tools keep everything you give them in one growing pile of memory and uploads. When many files look alike, the AI can pull a detail from the wrong one. The fix is organizing the firm's information so each task reads only the files for that matter.

Is it safe to upload client documents to an AI chat tool?

It depends on the account, its terms and its settings. ABA Formal Opinion 512 says the lawyer has to know whether the tool retains or trains on client information, and needs informed consent before using one that could disclose it. Business-tier accounts with training excluded are the right start, with someone checking the terms and supervising use.

Will a newer AI model fix drafts that got worse?

Usually not. The problem is the information the model is given. A bigger model reading the same disorganized pile makes the same kind of mistakes.