Synset research

When a verified source field is enough

Synset tested how an unchanged MedGemma 1.5 4B system delivered gene answers when the requested source evidence was available or inadequate. A fixed source-verification rule reduced explicit unsupported gene answers from 29/60 to 0/60 on inadequate-evidence cases. Correct answers with the verified source stayed at 60/60. A deterministic reader of the admitted gene-symbol field matched the guarded delivery on all 120 tasks, using zero model calls. For this structured task, the measured behavior can be delivered by checking the source and reading its field.

Unsupported assertions on inadequate evidence and preserved correct answers on verified evidence, with confidence bounds.

Buskila’s repeated-generation study on MedQuAD motivated the audit of answer stability. It included MedGemma 1.5 4B and measured how much answers varied across repeated questions. Synset’s Orphadata experiment asks a different question with a directly checkable answer: does the supplied record support the gene symbol the system delivers? The 29 unsupported answers and the effect of verification were measured in this experiment. Buskila, Evaluating Small Open LLMs for Medical Question Answering, arXiv preprint (2026)

What the comparison tested

The independent evaluation used 60 held-out disease/gene/publication groups from the English July 2026 Orphanet Knowledge Base, with one verified and one inadequate input per group. The caller supplied the evidence. The positive condition included the typed, curator-assessed causal relation and its native gene symbol. In the negative condition, 20 groups received no source hit, 20 a different-family passage, and 20 a question-only passage. The rule required the requested family, record, edition and admissible passage to match before delivering an answer. Inadequate evidence yielded UNKNOWN. The deterministic comparator inspected the same trusted request and passage and had no access to the evaluation key. Orphadata gene associations

Of the 60 inadequate inputs, the unguarded model produced 29 explicit unsupported assertions, 28 correct responses and three unscorable responses. The unsupported assertions were distributed 11, 0 and 18 across the three negative conditions in the order above. All planned cases remained in the denominators. The guarded system and lookup each returned the expected response on all 60 positive and 60 negative cases.

Uncertainty and interpretation

The one-sided 95% lower bound on paired corrections minus possible harms was 29.27 percentage points, above the required 25 points. The calculation subtracted an exact upper bound on worsening from an exact lower bound on correction, each at one-sided 97.5%. On the 60 positive families, the one-sided exact upper bound was 4.87% for new loss of correctness and separately 4.87% for loss of coverage, both below the 5% margin. Three unscorable baseline answers received no benefit credit.

The experiment changed answer delivery and made zero weight updates. The caller-bound source task establishes neither autonomous retrieval performance nor improvement in free-text clinical reasoning. An earlier, separate MedQuAD evaluation found fewer unsupported assertions but left preservation unresolved on 37 families. We report the two evaluations separately.

Run the source verifier

Download the Python implementation and the synthetic input. Run python source_verification.py synthetic_example.json with Python 3.10 or later. Both routes return ["gene", "EXAMPLE42"] for the fictional example. Remove the retrieved passage or supply one with a mismatching binding or digest to get ["unknown", null].

The standalone file carries the exact evaluated passage verifier, answer reader, and deterministic symbol lookup, with a new command-line wrapper. The caller supplies an independently admitted expected binding with family, source_record_id, edition, passage_id, and the SHA-256 of the admitted UTF-8 passage. The retrieved object supplies the same identity fields and the exact passage text. The guarded route also accepts pipeline_answer, the final-answer string from the calling model interface. The lookup route reads a singular Symbol: ... line from verified text and requires no model or base checkpoint. A verified passage with missing or ambiguous symbol fields yields an unscorable lookup rather than a guessed answer.

The caller owns source admission. Build the expected binding from trusted source records before retrieval and independently of the returned hit. The verifier checks identity and passage integrity. It leaves the curator-assessed relationship and edition in the caller’s source contract, and it does not perform clinical adjudication.

Data and figure

Source data are attributed to Orphanet / Inserm, Orphadata Science, July 2026 release, English gene associations dated 23 June 2026, available under CC BY 4.0. Synset constructed the comparison and reports its own aggregate evaluation here. Run python render_figures.py with NumPy and Matplotlib to redraw the figure. Run python verify_bounds.py with SciPy to reproduce the three confidence bounds from paired counts. Aggregate results · Original data · CC BY 4.0

Companion MedGemma weight adaptation · MedNLI methods and research code