In our new blog series, Delphyr Engineering, our engineers share practical insights from building AI systems for real clinical use. In this first piece, Tim de Boer,  explores a foundational question in medical AI: how can clinicians trust AI-generated guidance? Drawing on hands-on experience, Tim explains why citations are not a “nice to have” but a requirement for safe medical AI, and why getting them right means going beyond simple document references to exact, verifiable source quotes.

Why trust in medical AI starts with citations

If a colleague offers clinical advice but can’t explain its basis, you’d likely want to understand the rationale or verify it before acting on it. The same standard applies to medical AI: when clinicians use AI for guidance on guidelines or patient data, the output must be traceable to reliable sources. 

Citations are essential, they allow professionals to verify information, assess its relevance, and make informed decisions. Without clear references, AI responses can’t be trusted in a clinical setting.

Trust requires traceability, not just retrieval

A common problem with AI models is that they can sound confident even when they are wrong. In clinical contexts, this is especially dangerous. A model might recommend a treatment, cite a guideline, or summarize patient data, yet the information could be incomplete, outdated, or simply incorrect. 

For example, a model might advise starting antihypertensive medication for a patient based on a single elevated blood pressure reading, claiming this is recommended by the guidelines. However, starting medication based on a single elevated reading alone is not consistent with these recommendations in the absence of compelling clinical circumstances. This phenomenon, known as AI hallucination (plausible-sounding outputs not grounded in evidence), undermines trust in medical AI.

Retrieval-Augmented Generation (RAG) was supposed to solve AI hallucination by grounding responses in real documents. It’s an AI approach that combines two steps to produce answers, and it’s especially important in medical AI:

1. Retrieval: before answering, AI searches a trusted knowledge base (such as clinical guidelines, research papers, or patient records).

2. Generation: the AI model then uses those retrieved documents to generate a response in natural language.

But, does it work? The answer is: partially. The AI model consults clinical guidelines, patient records, and research papers before answering. But without citations, you still have a black box. AI gives you an answer. You know it consulted documents. But which parts of the answer came from which documents? Is it accurately representing the sources, or subtly distorting them? When AI says "according to the diabetes guidelines," which specific guideline section does it mean?

Healthcare professionals shouldn't have to guess. They shouldn't have to manually search through source documents to verify every claim. AI needs to show its work. Citations can solve this. But not all citations are created equally.

Beyond document references: exact quotes

Many AI tools cite at the document level: "This came from Source X." That's better than nothing, but it's not enough for medical AI. Think about it: a clinical guideline might be multiple pages long. Telling a physician "this recommendation comes from guideline source X" helps them only a little in verifying the claim. They'd need to read the entire document to find the relevant passage. Instead, adding the exact source snippet used by the model greatly simplifies physician verification by eliminating the need to search through the document and pointing directly to the relevant section. 

```
LLM Claim: Adults with type 2 diabetes should be supported to reach HbA1c targets of 48 or 53 mmol/mol, depending on hypoglycaemia risk.

Cited source passage: “Support adults with type 2 diabetes to reach and maintain their HbA1c target… aim for an HbA1c level of 48 mmol/mol (6.5%)… [or] 53 mmol/mol (7.0%).” Type 2 diabetes in adults: management NICE guideline, section 1.6
```

This changes everything about how clinicians can evaluate and trust AI output. Instead of a vague reference to a long document, AI is forced to be precise and transparent about the exact evidence behind each claim.

• Precision: AI can't vaguely gesture at a document. It must identify the specific sentence supporting its claim.

• Instant verification: Clinicians see the actual source text alongside AI's interpretation. They can verify accuracy in seconds, not minutes.

• Accountability: When something goes wrong, there's a clear audit trail. We know exactly what the AI model read and what it concluded.

• Trust: Seeing the evidence alongside the claim builds confidence. Users learn to trust the system because they can verify it.

How Delphyr generates citations

We teach our LLM to generate citations inline as it writes its response. As the model produces text, it embeds citations naturally at the point of each claim. The model learns that every statement derived from a document must be cited. Not later. Not approximately. Right there, with the exact snippet.

This approach has a key advantage: AI only makes claims it can support. Because citation happens during generation, not after, the model develops an internal consistency. It won't make a claim and then struggle to find supporting evidence: it finds the evidence first.

Accuracy over speed

One important trade-off is that adding citations during generation slows responses. Generating citations inline and validating them takes time. We could make the system faster by skipping citations. Or by citing less rigorously. Or by adding citations after answer generation, by letting the model generate an answer first and then linking that answer back to source passages afterward. 

We don't.

Post-hoc citations don’t actually solve verification. Once a claim is generated, there is no objective way to prove that a retrieved source truly supports it. Similarity is subjective. Using exact quotes removes this ambiguity entirely: the snippet either appears verbatim in the source or it doesn’t. Validation becomes a binary check.

That ambiguity pushes verification back onto the user.

Think about the alternative: a physician gets a quick answer, but with loosely attached citations. Now they spend two minutes manually searching through guidelines to determine whether those sources really support the claim. You've traded one second of system latency for two minutes of human labor, and introduced verification errors. Or worse: the physician doesn't verify the loosely attached sources at all (automation bias), as it would take too much time. They trust AI because it's fast and confident. And that's when mistakes happen.

Citations shift the verification burden from humans to systems, but only if they can be validated deterministically. With our approach, the LLM takes longer to respond, but the response comes pre-verified. Clinicians can trust it immediately, or verify it in seconds by checking the cited snippets.

What we've learned so far

As we’ve been developing citation systems for medical AI at Delphyr, we’ve learned several important lessons about what works (and what doesn’t) when trying to make AI outputs reliable and verifiable.

Post-hoc citations are difficult to validate 

We explored validating claims made in the LLM responses by adding the citations post-hoc, but we found that approach is too ambiguous. Using exact quotes removes this ambiguity entirely. 

Models can be trained to cite reliably

Generating high-quality citations is not an unsolved problem. With the right training setup, models can learn to consistently ground their claims in sources. When citation is a first-class requirement rather than an afterthought, the model adapts its behavior. It learns when to answer, when to abstain, and how to phrase claims so they can be directly supported by evidence. In practice, this works remarkably well.

Citations improve grounding

We have seen that forcing AI to cite every claim within its answer makes it more conservative and precise. The model learns to only make claims it can support with exact quotes. That's exactly the behavior you want in medical contexts.

Latency is acceptable when accuracy is guaranteed

The key is making the wait feel worthwhile: show the user the found sources, citation snippets, and explain what's being checked.

The bottom line

RAG without citations is a half-solution. It grounds AI responses in documents, but doesn't show which parts came from where. Citations, especially exact quote citations, complete the picture. They enable verification, build trust, and shift the burden of accuracy from healthcare professionals to the AI system.

The cost is latency. Adding citations and validating them in real-time takes time. But in medical AI, accuracy matters more than speed. Healthcare professionals need systems they can trust, not merely fast ones.

Because when patient care is on the line, ‘trust me’ will never be good enough.

Follow our journey in medical AI

Interested in how we're building reliable AI for healthcare? Follow our engineering blog for insights on medical AI infrastructure, evaluation frameworks, and production lessons learned.