TyKaoz, the macOS app I develop, turns a folder of documents into a wiki: a model reads each source, then creates or extends pages (one per person, company or project) with writing tools. You then ask the wiki questions, and every answer cites the pages it comes from. All of it can run on the Mac, on MLX.
The question I get most often is a simple one: I have a 16 GB Mac (or 24, or 36), which model should I pick? Rather than answer by feel, I built a test bench and put the models through it one by one.
The limits first
These figures hold for one corpus: 32 documents in French, about a fictional company. Each model made one run per reading order of the sources; the verdicts come from an LLM judge; the answers cover 40 questions, and on 40 questions a gap of a few points is noise (the 95% confidence interval is about ±15 points). Speeds were measured on a single machine, a 36 GB M3 Pro. Read what follows as an order of magnitude for document work in French, not as a general ranking of models.
The test bench
The corpus tells the story of a fictional Breton tidal-energy SME over several years: meeting minutes, press articles, emails and reports. It holds 23 entities to find, 110 reference facts, and ten traps: five contradictions between sources (two documents give two different values) and five supersessions (a piece of information replaces an older one: a departure, a new headcount). A model that builds a reliable wiki must flag the former and update for the latter, without losing anything else.
Two tests:
- Ingestion: the model reads the 32 sources one by one and writes the wiki. We measure entity recall, fact loss (a fact present at some point that is gone at the end), contradictions flagged, supersessions applied, and order sensitivity: we run the ingestion again in reverse order, then count the facts that differ between the two wikis.
- Answers: we freeze one wiki, the one built by the best model, and each candidate answers 40 questions of six kinds: simple, multi-step, synthesis, dates, about a contradiction, and with no answer in the wiki (where it must say so rather than make something up). Forty more questions are held out for the final report, without having been used to tune anything.
The judge is Mistral Small 4 in NVFP4, served by vLLM on my DGX Spark (the setup from this article, whose quality I measured here). Before trusting it, I annotated 30 verdicts from each test myself: it agrees with me on 28 of the 30 facts, and on all 30 answers. When it says a fact is present, it must quote word for word the passage of the wiki that shows it, and the quote is checked.
My own measuring tool was wrong
A first pass showed fact losses of 6 to 12% for every model, the best ones included. Going through the cases one by one, I found that more than half of these losses came from the grader, not the models:
- a positive verdict whose quote was copied badly (two passages glued together, a title kept in brackets) turned negative;
- the excerpts shown to the judge left out the page title: on a person's page, the paragraph describing their role without repeating their name was never shown;
- text normalisation stripped non-ASCII characters, so a non-breaking hyphen welded two words together and the quote could not be found;
- two reference facts were cut off at the
#character, which YAML reads as a comment.
Once these defects were fixed, the losses of the best models dropped to 0 or 1%. Two traps were also badly framed and were rewritten. The judge's calibration did not move. If you evaluate models with an LLM judge, read your failures by hand: mine said as much about my code as about the models.
Ingestion: no local model passes
The thresholds are the ones I set for a wiki worth trusting. "Order" is the share of facts that differ between the two reading orders.
| Model | Recall | Loss | Contradictions | Supersessions | Order |
|---|---|---|---|---|---|
| Threshold | ≥ 85% | ≤ 2% | ≥ 4/5 | ≥ 4/5 | ≤ 5% |
| Mistral Small 3.2 4‑bit Mac, 13 GB | 83% | 2% | 2/5 | 4/5 | 26% |
| Gemma 4 26B 4‑bit Mac, 14 GB | 61% | 0% | 1/5 | 1/5 | 23% |
| Mistral Small 3.2 6‑bit Mac, 18 GB | 52% | 5% | 1/5 | 3/5 | 30% |
| Qwen 3.6 35B 4‑bit Mac, 20 GB | 83% | 6% | 2/5 | 3/5 | 23% |
| Gemma 4 26B 8‑bit Mac, 26 GB | 91% | 0% | 1/5 | 1/5 | 26% |
| Mistral Small 4 Switzerland | 78% | 5% | 2/5 | 2/5 | 43% |
| gpt‑oss‑120b France | 87% | 1% | 3/5 | 2/5 | 11% |
| Qwen 3.5 397B France | 87% | 0% | 5/5 | 5/5 | 4% |
The memory shown is the peak measured during ingestion. The "Switzerland" and "France" models are hosted by European providers. As an out-of-competition reference, Claude gets 83% recall, 0% loss, 4/5 and 4/5.
Only Qwen 3.5 397B clears every threshold. The local models cover the facts well enough (Gemma 26B in 8-bit even finds more entities than it does), but they fail where a wiki differs from a pile of notes: they rarely flag contradictions, rarely apply supersessions, and a quarter of their facts change with the order the documents are read in. Your wiki would say something else if you had dropped your files in a different order.
Mistral Small 3.2 in 4-bit is the local model closest to the mark. In 13 GB, it finds as many entities as Qwen 3.6, loses few facts and applies four supersessions out of five, right at the threshold. But it flags only two contradictions, and a quarter of its facts still depend on the reading order.
Quantisation weighs heavily: Gemma 26B goes from 61% to 91% recall between 4 and 8 bits. It changes nothing, however, in how contradictions are handled. And more bits do not always help: Mistral Small 3.2 does worse in 6 bits than in 4 on every measure. It writes less (162 attempted writes against 277) and leaves nearly half the entities without a page. On a single run, part of this gap may be chance; the direction is clear.
On speed, on the M3 Pro, Gemma 26B ingests a source in under two minutes, Qwen 3.6 in seven, because it reasons at length before writing, and Mistral Small 3.2 in eight to nine. Among the hosted models, gpt-oss-120b covers almost as well as Qwen 3.5 397B for about a quarter of the price, but does not check for contradictions.
Answers: a 48 GB Mac is enough
Ingestion happens once per document. Questions come back every day. If the wiki is built by a large model, which local model can answer from it? Here, all of them answer from the same wiki, the one built by Qwen 3.5 397B.
| Tier | Model | Peak memory | Correct answers | Time per question |
|---|---|---|---|---|
| 8 GB | Gemma 4 E4B 4‑bit | 4.4 GB | 32% | 9 s |
| 16 GB | Gemma 4 E4B 8‑bit | 7.8 GB | 25% | 8 s |
| 24 GB | Gemma 4 26B 4‑bit | 14.4 GB | 48% | 22 s |
| 24 GB | Mistral Small 3.2 4‑bit | 12.8 GB | 45% | 37 s |
| 32 GB | Qwen 3.6 35B 4‑bit | 19.6 GB | 60% | 57 s |
| 48 GB | Gemma 4 26B 8‑bit | 26.1 GB | 75% | 22 s |
The tier is the smallest Mac configuration where the measured peak leaves enough room for the system and other apps. All these models ran on the same 36 GB machine: quality does not depend on the machine, speed does.
Gemma 4 26B in 8-bit gets every simple question right (10 out of 10) and every question about a contradiction (3 out of 3), with a little more trouble on dates (5 out of 6). It weakens on multi-step and synthesis questions (4 out of 7 each), where it often misses part of the expected answer, and on questions with no answer (4 out of 7), where it sometimes fills the gap instead of saying it does not know.
Qwen 3.6 does worse and more slowly: it reasons for a long time before each answer. Below 24 GB, one correct answer in three is not enough to base a decision on the wiki. The gap between the two Gemma E4B variants, on the other hand, is within the noise.
At 24 GB, Mistral Small 3.2 and Gemma 26B are on a par: three points apart is noise. Mistral almost always cites a page it was actually shown (29 times out of 30), but it fails all three questions about a contradiction: it gives a single value where the wiki holds two.
Before blaming the model
Before the full night, each model goes through a pre-filter: ingesting the first nine sources. Mistral Small 3.2 first scored 0% there. It had written no page, because the chat template shipped with its MLX conversion described tool calls in a format older than the one the model was trained on. Its calls were never recognised. Once the template was rewritten in the right format, it finds three quarters of the facts or more at the same stage (74 to 83% depending on the run).
Its first full night revealed a second defect, this time in TyKaoz. Mistral ends its lines with two spaces, Markdown's line break, including the --- line that opens a page's header. It sometimes also writes the first field on that same line, or even the whole header. The app did not recognise these variants: the page lost its title, and the write was refused. Three fixes later, its recall goes from 70% to 83%, and the share of facts that depend on the reading order from 40% to 26%. The bench was meant to evaluate models; it also found bugs in my app.
Apertus 8B, for its part, still writes no page, but for another reason. It does call its tools, but it encodes a multi-line page badly in their arguments: the line breaks disappear, the page header becomes unreadable, and every write is refused. That one is a limit of the model.
MiMo-7B-RL, a candidate for 8 GB Macs, also stops at the pre-filter, with 22% of the facts. It is a reasoning model, and it reasons for so long that it reaches the 4,096-token limit of a turn before calling a tool: five sources out of nine produced no page, and the pages it did write hold placeholder text instead of facts.
Before concluding that a model is weak at a tool task, check that it receives the tools in the format it knows, and that your own code reads what it writes.
What to take away
- To build the wiki, no model that fits on a 48 GB Mac is enough: you need a large model. In this bench, only Qwen 3.5 397B manages it. gpt-oss-120b is a cheaper option if you accept checking contradictions yourself.
- To ask it questions, a 48 GB Mac with Gemma 4 26B in 8-bit gives three correct answers out of four, in about twenty seconds, without a single question leaving the machine.
- At 24 and 32 GB, it is usable with care: one correct answer in two, checking the cited sources. At 24 GB, Mistral Small 3.2 in 4-bit and Gemma 4 26B in 4-bit answer equally well; Mistral builds a better wiki, but four times more slowly.
- At 8 and 16 GB, the local wiki is not reliable today, and MiMo-7B-RL does not change that. These models remain useful for conversation, not for basing a decision on your documents.
Two measurements are still missing. Mistral Small 3.2 in 8-bit passes the pre-filter, but on my 36 GB Mac its full night runs into memory: hours on a single source, up to the ten-hour limit, with swap nearly full. It will have to be measured on an actual 48 GB Mac. The 128 GB tier on the DGX Spark also remains, with gpt-oss-120b served locally. I will update this article with their figures.