Answer quality
Each change is tested on the same 50 questions, each with a reference answer. 12 of them are not answered by the documentation, and on one of them two documents contradict each other. A model grades every answer, and code counts invented answers and missing sources.
Current configuration · prompt v2
Mistral ministral-14b-2512
Details →Overall score
95.0
out of 100, graded by Gemini
Invented answers
0%
on the 12 undocumented questions
Missed answers
5%
marked “not documented” by mistake
Expected sources cited
91%
0 blocked for lack of a source
Gemini gemini-3.5-flash-lite
Details →Overall score
Pending
not graded by Gemini yet
Invented answers
0%
on the 12 undocumented questions
Missed answers
3%
marked “not documented” by mistake
Expected sources cited
94%
0 blocked for lack of a source
Grader calibration
13 answers were also graded by hand, mostly ones where the two graders disagreed. The question was simple: could this answer go to the customer as is? The scores above use the grader that agrees most with the person.
| Evaluator | Agrees with the person | Too lenient | Too strict | Gradings compared |
|---|---|---|---|---|
| Ministral 14B as grader | 13% | 0 | 13 | 15 |
| Gemini as grader | 94% | 1 | 0 | 17 |
| Deterministic rule (abstention) | 85% | 1 | 1 | 13 |
Prompt versions
Each version fixes errors found in the weakest answers of the previous one. v2 is in use because it is the only one where neither model invented an answer. v3 scores higher but invents one again.
| Prompt | Mistral (graded by Gemini) | Gemini (graded by Mistral) | ||||
|---|---|---|---|---|---|---|
| Score | Invented | Missed | Score | Invented | Missed | |
| v1 | 96.1 | 8% | 0% | 88.4 | 17% | 0% |
| v2in use | 95.0 | 0% | 5% | 90.5 | 0% | 3% |
| v3 | 97.7 | 8% | 3% | 87.7 | 8% | 0% |
Model × grader matrix · prompt v1
Both answer sets are graded by both models. Only compare scores in the same column, since graders are not equally strict. Neither grader favoured its own model.
All runs
| Run | Drafted by | Graded by | Prompt | Retrieval | Score | Invented | Missed | Sources |
|---|---|---|---|---|---|---|---|---|
| 2026-10-27 11:39 | Mistral | Mistral | v2 | embedding · k=5 | 87.8 | 0% | 5% | 91% |
| 2026-10-27 11:35 | Gemini | Mistral | v3 | embedding · k=5 | 87.7 | 8% | 0% | 96% |
| 2026-10-27 11:30 | Mistral | Gemini | v3 | embedding · k=5 | 97.7 | 8% | 3% | 93% |
| 2026-10-27 11:25 | Gemini | Mistral | v2 | embedding · k=8 | 86.9 | 8% | 3% | 93% |
| 2026-10-27 11:20 | Mistral | Gemini | v2 | embedding · k=8 | 95.8 | 0% | 5% | 93% |
| 2026-10-27 11:15 | Gemini | Mistral | v2 | embedding · k=5 | 90.5 | 0% | 3% | 94% |
| 2026-10-27 11:08 | Mistral | Gemini | v2 | embedding · k=5 | 95.0 | 0% | 5% | 91% |
| 2026-10-27 11:03 | Gemini | Gemini | v1 | embedding · k=5 | 95.1 | 17% | 0% | 98% |
| 2026-10-27 10:59 | Mistral | Mistral | v1 | embedding · k=5 | 87.1 | 8% | 0% | 96% |
| 2026-10-27 10:58 | Gemini | Mistral | v1 | embedding · k=5 | 88.4 | 17% | 0% | 98% |
| 2026-10-27 10:53 | Mistral | Gemini | v1 | embedding · k=5 | 96.1 | 8% | 0% | 96% |