Original sources
-
SLM Finetuning for Natural Language to Domain Specific Code Generation in Production
A fine-tuned Mistral-7B replaced GPT-4 with retrieval for generating workflow code: similar quality, about 10× lower latency.
Opens on arXiv (not licensed for re-hosting)Source: arXiv 2604.09952 -
Fine-Tuning Small Language Models for Function-Calling: A Comprehensive Guide
Step-by-step Azure tutorial for LoRA-fine-tuning Llama-3.2-3B to call functions, including trimming function descriptions from the prompt.
Opens on Microsoft Tech Community (web article, copyrighted)Source: techcommunity.microsoft.com -
Unveiling the Secret Recipe: A Guide for Supervised Fine-Tuning Small LLMs
Hyperparameter study for full fine-tuning of 3–7B models: large batches, low learning rates, no warm-up, all data in one run.
Earlier papers from the same Microsoft team
-
A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation
The GPT-4 + retrieval baseline used by the 2026 paper. Fine-tuned Codex scored best on similarity; tuned retrieval matched it.
-
Plan with Code: Comparing approaches for robust NL to DSL generation
Same comparison, framed as workflow automation in robotic process automation (RPA).
Small models for function calling
-
Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
Zero-shot, few-shot and fine-tuned small models compared, with latency and memory measured on an edge device.
-
TinyAgent: Function Calling at the Edge
Fine-tuned 1.1B and 7B models that match GPT-4-Turbo at function calling, running locally on a MacBook.
Opens on arXiv (not licensed for re-hosting)Source: arXiv 2409.00608 -
Optimizing Small Language Models for In-Vehicle Function-Calling
Phi-3 mini pruned, quantised and fine-tuned to run in a car at 11 tokens per second without a GPU.
-
TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices
Small models on the Berkeley Function Calling Leaderboard; 1–3B models clearly beat models under 1B.
-
Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning
Fine-tuned OPT-350M reports a 77.6% ToolBench pass rate; check the evaluation setup before relying on it.
LoRA versus full fine-tuning
-
LoRA Without Regret
LoRA matches full fine-tuning when applied to all layers with enough rank; needs about 10× the learning rate and smaller batches.
Opens on thinkingmachines.ai (web article, copyrighted)Source: thinkingmachines.ai -
LoRA Without Regret (TRL documentation summary)
Hugging Face's reproduction and summary of the Thinking Machines findings, with TRL settings.
-
LoRA Learns Less and Forgets Less
On code and maths, LoRA learns the target task worse than full fine-tuning but keeps more of the model's other abilities.
-
LoRA vs Full Fine-tuning: An Illusion of Equivalence
LoRA adds new dominant directions ("intruder dimensions") to the weights, which cause forgetting.
Fine-tuning versus retrieval
-
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
For factual knowledge, retrieval beats fine-tuning on raw text; paraphrased variations help models learn new facts.
Production and the case for small models
-
LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
310 LoRA models across 31 tasks beat GPT-4 by 10 points on average; 25 adapters served from one A100.
-
Small Language Models are the Future of Agentic AI
Position paper: most calls inside AI agents are narrow and repetitive, so small models are enough and cheaper.