Article
We Tested Five AI Models on a Confidential Report
This was neither a financial report nor a spreadsheet. The source was a 13,708-character internal research report split into 12 thematic sections. A decision-maker did not need a shorter retelling. They needed a concise map of barriers, motivations, and trust issues, with every finding backed by an exact quote. The obvious approach would be to drop the DOCX into Claude or Codex. But what if that is not allowed? For this experiment, sending the source document to external services was undesirable. This is not a universal ban. Whether cloud models are acceptable depends on company policy, contracts, and the category of data involved. In our case, the document stayed on the laptop. Claude and Codex did not read or analyze it, but they helped us set up the local pipeline: write the scripts, define the output schema and checks, and assemble the resulting files. We ran five AI models on a MacBook Air M5 with 32 GB of memory. We were not testing abstract intelligence. We were testing a practical task: produce decision-ready material, substantiate every finding with the source text, and keep the data inside the chosen environment.
01
What we wanted the models to produce
The research answered practical questions: whether customers notice the relevant product section, where they abandon the process, how they choose a supplier, why they trust or distrust the offer, and which acquisition channels lead to a purchase. We did not need a brief summary. We needed an analytical memo.
Each model had to identify five to nine distinct findings, group them into four to six themes, and attach an exact quote from the DOCX to every finding. It also had to preserve the study's caveats and avoid turning observations from a limited study into claims about all customers.
- 01Read DOCX13,708 characters and 12 thematic sections
- 02Identify themesBarriers, trust, choice, and acquisition channels
- 03Extract findingsFive to nine distinct observations
- 04Verify the evidenceEvery finding has a verbatim quote from the source
The goal was not to shorten the text. It was to produce material for a decision that could be checked against the source.
02 — Spoiler
It works, but not as a chat session
Yes, a confidential report can be analyzed locally. The best run produced five thematic sections and eight findings. All eight supporting quotes matched the source document verbatim. The analytical step took just over eight minutes.
The useful result, however, was not a chat response. It was the whole pipeline. The model returns a strict structure. Code validates the schema and every quote. The verified data is then assembled into an HTML report that opens in a regular browser, with a short summary at the top and themes, findings, and exact evidence below.
JSON is not for the reader. It is the machine-readable structure for the next automation step, with themes, findings, and references to quotes. React matters only when the same verified material needs to become an internal portal or dashboard with cards, navigation, and charts. The anonymized text copy came from a separate safeguards test. These are not four versions of the same run.
HTML in browser
Verified HTML report
Brief summary
People postpone the decision when they cannot compare the terms quickly.
React in internal portal
Interactive representation
Research panel
The card shows the theme, verification status, and a link to the original fragment.
Anonymized mockups show the result structure, not the content of the original report. JSON and HTML were generated in the main run. React and anonymization were tested separately.
03
How the experiment was structured
The source was a 13,708-character DOCX containing a complete internal research report with 12 thematic sections, text, data, and numerical indicators. We do not disclose the company, the product, or the content of the research.
Every model received the same text and the same output contract: identify five to nine findings, group them into four to six sections, and attach an exact source quote to every finding.
The test ran on a MacBook Air M5 with 32 GB of memory. Generation temperature was set to zero and the seed to 42. We made one deterministic run for each condition, so the figures show direction rather than statistically stable results.
- DOCX
- Format of internal report
- 12
- Thematic sections in the research
- 13 708
- Characters in the document
- 5
- AI models compared
MacBook Air M5, 32 GB memory. The original document did not leave the local environment.
04
Which models were run
Four models ran through Ollama 0.30.11. Muse Glimmer ran through LM Studio 0.4.16. The MLX models did not make the final comparison: Qwen 3 14B did not complete the request under the current memory load, and Devstral 2507 was not rerun on the new document. GPT-OSS 20B in LM Studio also failed to complete the request and received no score.
The chart shows the size of the quantized weights or model package, not peak memory use. Generation also needs memory for context, cache, and runtime data, so the actual load is higher.
Ollama used Q4_K_M, for Qwen 3.5 it was Q4_K. Muse Glimmer worked in k-quant GGUF format.
05
How the models' profiles differed
One report shows only part of the picture. The radar chart below comes from a broader field test across eight types of tasks, including code, document analysis, quote accuracy, building a React interface, agentic actions, and proposing an anonymization workflow.
There was no universal winner. Muse Glimmer analyzed the report best and preserved the evidence trail, but lagged in agentic tasks. Qwen 3 4B was noticeably faster and, after one worked example, built a valid React presentation. Devstral, Gemma, and Qwen 3.5 looked stronger in agentic scenarios than in work that required exact quotes.
The effect of context was especially clear. One good example materially improved the React result for two models. The choice is not only about the model. It is also about how the task is specified.
Diagram rendered with Pele, the diagram engine used by Markdown editor Mark.
Scale from 0 to 5. Each model was run once in each scenario, so the numbers show direction, not exact ratings.
06
The largest model provided the best analytics
Muse Glimmer 30B produced the best analytical brief, scoring 4.6 out of 5. The small Qwen 3 4B came in second with 4.05. Devstral Small 2 and Gemma 4 scored 3.65 each. Qwen 3.5 Abliterated got 3.15.
This chart alone does not identify a winner. The gap between first and second place is clear, but quality has to be considered together with evidence accuracy and time. That is where the comparison becomes more useful.
Evaluated for completeness of summary, absence of repetition, structure, caution in conclusions, and readability.
07
Exact quotes changed the results
Muse Glimmer was the only model to pass the strict evidence check. All eight of its quotes appeared verbatim in the document. Qwen 3 4B passed five of eight, Devstral Small 2 five of nine, Gemma 4 four of seven, and Qwen 3.5 Abliterated two of seven.
These errors are easy to miss in a normal interface. A quote looks plausible, fits the theme, and may differ from the original by only a few words. But then it is no longer a quote. If that material reaches a decision-maker or a client, a polished paraphrase starts masquerading as evidence.
Scores are based on the share of claimed quotes found verbatim in the source document.
08
Why eight minutes per report is reasonable
Muse Glimmer prepared the best document in 8 minutes 7 seconds. Devstral Small 2 finished almost in 9 minutes. Qwen 3 4B completed in 1 minute 29 seconds and took second place in quality of summary.
For one substantive report, eight or nine minutes is not a long wait. The output is already a structured draft, but the model has not replaced the analyst. A person still has to check the meaning of the findings, the limits of the sample, and the final recommendations.
Speed starts to matter when there are dozens or hundreds of documents, or when an answer is needed interactively. Then a difference of almost 5.5 times becomes a real difference in throughput. For one important report, quality and evidence accuracy matter more.
One run on one machine. Time shows the order of magnitude, not guaranteed performance.
09 — Bonus
Can we anonymize the report and give it to Claude or Codex?
What if we anonymize the report, send it to Claude or Codex, and restore the original values afterward? Sometimes that can be acceptable, if company policy, contracts, and the data category allow it. We therefore tested separately whether local models could help with anonymization.
First, we tested the model on its own. It had to find sensitive fragments and replace them with reversible markers. The synthetic document contained 15 protected fragments. No model on its own met the zero-leak requirement.
That result is unsurprising. The instruction to find all sensitive data does not turn a probabilistic model into a firewall. It may find almost everything, but one missed fragment is enough for a leak.
For the real-document test, we deliberately narrowed the safeguards to numbers, ranges, and percentages. The model proposed replacements. Deterministic code then searched again for the same data types, filled in anything the model had missed, and only then allowed further processing. The mapping between markers and original values stayed in local storage.
This test does not show that the same method automatically protects names, document numbers, or other personal data. Those categories need their own rules and tests.
- 01Local analysisModel proposes replacements for numbers, ranges, and percentages
- 02Deterministic checkCode searches again and fills in anything the model missed
- 03Anonymized materialOnly text without remaining digits proceeds
- 04Local restorationOriginal values are restored from a private marker map
In this experiment, the deterministic check covered only numbers, ranges, and percentages.
10
Both models missed one number on the real document
For an additional check, we asked Qwen 3 4B and Muse Glimmer to find seven unique numerical values in the real confidential document. Both models correctly found six of seven, and each response also contained one invalid item. This was not a perfect result, even though the valid values themselves were reproduced with 100% accuracy.
The deterministic check added the missing value. After that, the anonymized text contained no digits, and the reconstruction restored every value exactly.
The strongest model did not solve the task alone. The smallest model showed the same result. Reliability appeared not from model size but from the process architecture.
There's no hard conclusion here. This scheme doesn't protect against leaks at 100% and guarantees nothing: the model might miss a fragment, and the code check only covers the data types it was designed for. Partial anonymization might be one step, but the decision to send the document to an external service cannot be made based on this test. It's determined by company policy, contractual restrictions, and specific data category checks.
- 7
- Unique values in source
- 6
- Found by each of two models
- 1
- Added by code check
- 0
- Digit leaks after the check
After local restoration, every replaced value was restored exactly.
11
Which scheme would we choose for work
If the goal is the best draft for an important analytical report on this machine, we would start with Muse Glimmer. Eight minutes for such a task are perfectly normal. If there are many documents and the result still passes automated and human checks, Qwen 3 4B looks more practical.
Devstral Small 2 was stronger in agentic scenarios than in evidence-heavy document analysis. Gemma 4 and Qwen 3.5 Abliterated may also be useful in other roles, but this test gave us no reason to choose them for confidential report analysis.
We would not let any model publish conclusions on its own or decide that the data is safe. The model prepares a draft. Code checks formal boundaries. A person makes the substantive judgment.
12
Conclusion
Local execution controls where the document is physically processed. It does not solve the quality problem or provide an automatic guarantee of confidentiality.
In our test, the larger model produced the best analytical material. The smaller model was several times faster and sufficiently good for a limited role. Both missed one sensitive number. Therefore, the working choice looks not like the name of one model but a combination of local execution, exact quotes, code checks, and clear human control.
The same principle, keeping the data inside your perimeter, runs through our speech and image-text projects: from meeting transcription to screenshot OCR on a local model.
- SpeechLocal-first voice assistantDictation and meeting notes on the laptop; speech never leaves the device.
- SpeechKnowledge base from chats and meetingsTranscription with speaker separation; decisions and tasks extracted and linked to their sources.
- OCRLocal vision model for agentsScreenshot analysis and OCR on a local model for coding and support agents.
- ArticleMeeting transcription without the cloud: Whisper and speaker separationHow to set up local meeting transcription: Whisper large-v3, pyannote for speakers, recognising familiar voices and extracting decisions tied to timecodes.