MEDFIT-LLM Explained: Fine-Tuning Small Language Models for Medical Tasks

The first entry in AI Research Explained — my own papers and other researchers' work, broken down in plain language.

The paper

MEDFIT-LLM: Medical Enhancements through Domain-Focused Fine Tuning of Small Language Models, co-authored with Arjun Jaggi and Sonam Naidu, published at the 2025 2nd International Conference on Research Methodologies in Knowledge Management, AI and Telecommunication Engineering (RMKMATE). It's picked up 5 citations since publication. Code, dataset splits, and the full evaluation harness are open source at github.com/adityak74/medfit-llm.

TL;DR: We fine-tuned four small open-weight LLMs (3B–9B parameters) on an identical synthetic healthcare Q&A dataset using LoRA on MLX, then scored base vs. fine-tuned on the same 50 held-out questions with the same weighted rubric. Model size did not predict outcome: the smallest model, Llama-3.2-3B, produced the largest overall improvement (108.2 points) and the most dramatic behavioral fix — it stopped degenerating into repeated sentences — while two larger 7B–9B models barely moved.

Why this is more than a single fine-tuning demo

Most public "I fine-tuned an LLM" write-ups test one model against one dataset and report a win. MEDFIT-LLM is a controlled, head-to-head study across four different model families under one identical protocol — same synthetic dataset, same LoRA recipe, same held-out validation set, same scoring rubric — which is what turns "fine-tuning helped" into a comparable, falsifiable result instead of an anecdote.

That distinction matters for the broader research question the field is actively arguing about: is parameter count the main lever for domain performance, or is it adaptation quality? Prior healthcare-chatbot work — a reliability study for Hajj-pilgrim health chatbots, a comparative sweep of BERT/DistilBERT/RoBERTa/DialoGPT/TinyLlama/GPT-2 for women's healthcare apps, and PharmaLLM's work on medicine-prescription chatbots — each demonstrated domain fine-tuning works, but largely as single-architecture case studies. MEDFIT-LLM's contribution is the controlled comparison: run the identical recipe across four architectures and let the divergence in outcomes tell you something about why fine-tuning works, not just that it does.

The problem it addresses

Large language models are good generalists, but medical text has its own vocabulary, abbreviations, and reasoning patterns that general training data underrepresents. The usual fix is to reach for a bigger model or heavier prompting — both of which raise cost and latency without addressing the underlying mismatch between training data and domain.

Small language models are cheaper to run and easier to deploy on constrained infrastructure, but out of the box some of them don't just ramble — as the results below show, one of them degenerated into repeating the same sentence verbatim until it hit the token limit. That's not a style problem, it's a reliability failure a healthcare chatbot cannot ship with. The open question we set out to test: how much of that gap closes with focused fine-tuning instead of raw scale?

Study design

We generated a synthetic dataset with Phi-4 rather than relying on scarce, expensive expert-annotated medical Q&A pairs — 10,000 healthcare-related question/answer pairs focused on the future impact of LLMs in healthcare. After deduplication, 6,444 unique questions remained, split into 5,155 training, 645 test, and 644 validation examples. Four small open-weight models — spanning three different model families and three quantization/precision configurations — were each LoRA-fine-tuned on MLX independently, then evaluated against their own un-tuned base checkpoint on the identical 50 held-out validation questions, using the identical prompts for base and fine-tuned runs.

The four models weren't chosen arbitrarily — they span three model families (Llama, Mistral, Qwen, Gemma) and three different precision/quantization setups (4-bit, full-precision, 8-bit), so the comparison stresses architecture and quantization as variables alongside fine-tuning itself, not just "does fine-tuning help" in isolation.

Full results

Every model moved on at least one axis, but no two models moved the same way. The full metrics, including question-repetition and bullet-point usage (not shown in the earlier summary), make that divergence explicit:

Direct-answer rate: the clearest signal

The most interpretable metric is how often a model led with a direct answer instead of general preamble. Llama-3.2-3B jumped 30 points; Mistral-7B jumped 20. Gemma-2-9B didn't move at all, and Qwen2-7B was already answering directly 42% of the time before fine-tuning and stayed there.

Mistral-7B got both more accurate and nearly 22% faster — the fine-tuned version cut 60–70% of the verbose preamble the base model produced, which is why generation time dropped even though nothing about the hardware or decoding settings changed. Llama-3.2-3B moved structurally in the opposite direction: fine-tuning pushed it toward numbered lists (+18%) and away from bold text (−34%) and bullet points (−24%), a shift toward the kind of organized, scannable formatting that reads well in a healthcare chat interface rather than dense markdown.

Overall improvement score: size didn't predict the winner

We combined direct-answer improvement, response structure, generation efficiency, and question-repetition reduction into a single weighted score per model. Llama-3.2-3B — the smallest model in the study, at 3B parameters — came out on top by a wide margin, more than the 7B and 9B models' scores combined.

That inversion — the smallest model improving the most — is the paper's central empirical claim: parameter count is not the primary determinant of fine-tuning payoff. What predicts payoff instead is how much headroom a base model has to begin with, and Llama-3.2-3B, as the next section shows, had the most obvious failure mode to fix.

The degenerate loop: what a 3B base model actually does wrong

Question-repetition reduction is the least glamorous metric in the table, but it's the most concrete evidence of what fine-tuning actually fixed. Asked "Can AI tools provide real-time feedback during clinical procedures to improve outcomes?", the base Llama-3.2-3B model produced this:

"AI tools can provide real-time feedback during clinical procedures to improve outcomes. AI tools can provide real-time feedback during clinical procedures to improve outcomes. AI tools can provide real-time feedback during clinical procedures to improve outcomes. AI tools can provide real-time feedback during clinical procedures to improve outcomes. AI tools can provide…" (continues verbatim until the token limit)

That's a degenerate repetition loop, not an answer — a known small-model failure mode, and one that's disqualifying for a product that has to hold a clinician's or patient's trust. The fine-tuned version of the same model, same prompt:

"Yes, AI tools can offer real-time feedback during clinical procedures. These tools use machine learning algorithms to analyze data from various sources such as images, videos, and sensor readings. They can help healthcare professionals identify potential issues or errors before they become critical, thereby improving patient safety and outcomes."

— followed by four concrete categories (computer vision, wearables, virtual assistants, robotics), each with a one-line explanation. Only Llama-3.2-3B showed any question-repetition change in the study (−8%); the other three models never repeated in the first place. That's the real story behind its 108.2 improvement score: fine-tuning didn't just make Llama-3.2-3B more accurate, it fixed a structural failure mode that no amount of prompt engineering reliably resolves.

Where fine-tuning didn't move the needle

A rigorous comparison has to report the null results too. Gemma-2-9B's overall improvement score was 12.0, and Qwen2-7B's was 1.8 — both fine-tuned on the identical dataset and recipe as Llama-3.2-3B and Mistral-7B. For Qwen2-7B specifically, base and fine-tuned responses were frequently near-identical in substance. Asked "Do AI systems help streamline electronic health record (EHR) management?", both versions opened with "Yes, AI systems can help streamline EHR management by automating tasks such as data entry, documenta…" — the fine-tuned model trimmed some verbosity but didn't change what it said.

The likely explanation: both Gemma-2-9B and Qwen2-7B were already well-aligned instruction-tuned checkpoints with no repetition problem and a reasonable direct-answer baseline (Qwen2-7B in particular was already answering directly 42% of the time). There was less broken behavior for the same LoRA recipe to fix. That's a meaningful negative result on its own — it suggests fine-tuning payoff correlates with how much headroom a base model has, not with how large it is.

Limitations and what's next

The paper is explicit about scope: this is a 50-question held-out validation set, one synthetic dataset generated by a single model (Phi-4), and one fine-tuning recipe (LoRA on MLX) applied uniformly across four architectures rather than tuned per-model. Two directions follow directly from that:

• Retrieval-augmented generation. Fine-tuning shapes how a model answers; it doesn't govern what facts it's allowed to answer with. Layering RAG on top would let a fine-tuned model stay current and grounded instead of relying on parametric memory for medical facts, which is the harder reliability problem in a domain that changes as fast as healthcare guidance does.
• Generalizability across subdomains and populations. A synthetic Q&A set built around one framing ("the future impact of LLMs in healthcare") doesn't establish whether the same fine-tuning payoff holds for other healthcare subdomains or across more diverse patient populations and phrasing styles — that's flagged directly as future work.

Why it matters

This connects directly to the local-first, infrastructure-minded thesis running through the rest of this site: you don't always need the biggest model — you need the right model for the domain, tuned well and run cheaply. A 3B model that stops degenerating and starts leading with the answer beats a larger model that never had that failure mode fixed for it, and LoRA fine-tuning on MLX makes that adaptation cheap enough to run on a laptop rather than a cluster. That's the same efficiency argument behind quecto and the rest of the infrastructure work on this site, applied to a domain — healthcare — where getting the answer wrong isn't just an inconvenience.

Banner photo by 8machine_ on Unsplash.

Related reading

• AI Research Explained — the section this explainer lives in, with more papers and experiments on the way
• What Is an AI Agent Harness? — the systems layer that decides how a fine-tuned model like this actually gets deployed and evaluated in production
• Field Notes — infrastructure-focused notes on memory, routing, and local inference