When DPO and LoRA Stop Helping — Lessons from GPT-2 Scale

Modern LLM recipes almost always mention the same ingredients: supervised fine-tuning (SFT), then Direct Preference Optimization (DPO), often with LoRA instead of full fine-tuning (FFT) to save memory. That stack is standard on 7B-and-up models—but what happens when the backbone is GPT-2 size and the data budget is modest?

An Empirical Study of SFT–DPO Interaction and Parameterization in Small Language Models (Feng and Yang, Stanford) runs that question as a controlled experiment. The short answer: parameterization matters more than adding a preference stage, and LoRA does not automatically buy speed on compute-bound hardware.

The Paper

Paper: An Empirical Study of SFT–DPO Interaction and Parameterization in Small Language Models (Feng and Yang)

Venue: arXiv preprint (March 2026)

Topic: SFT, DPO, LoRA, small language models, alignment pipelines.

Code: github.com/Harry20030331/cs224n_project

The Problem and Why It Matters

Large-model practice has outpaced small-model evidence. Teams routinely copy SFT→DPO pipelines and PEFT defaults from billion-parameter setups without checking whether those choices still pay off when capacity and data are both tight.

That gap matters for education, edge deployment, and lab settings where a 124M-parameter decoder is still the workhorse. Course projects, ablation studies, and domain-specific adapters on small backbones need guidance that is not extrapolated from Llama-scale runs.

The authors isolate two axes on a GPT-2 (124M) decoder:

1. Training recipe: SFT-only, DPO-only, or staged SFT→DPO (with handoff timing and hyperparameter sweeps).
2. Parameterization: FFT vs. LoRA (rank sweep 4 / 8 / 16).

Tasks span classification (Quora Question Pairs paraphrase detection, up to 283k examples) and generation (131 Shakespeare sonnets, chrF on continuations).

The Method

SFT minimizes standard cross-entropy on labels or continuations. DPO prefers chosen over rejected outputs with the usual Bradley–Terry-style loss; β and learning rate are swept on paraphrase detection starting from the best SFT checkpoint.

For paraphrase detection, preference pairs are simple: correct label vs. incorrect label on the same input—so the preference signal closely mirrors supervision. For sonnets, chosen text is the gold continuation; rejected text is a model sample filtered by chrF band (strategies V1–V3 vary overlap with SFT data and pair count).

All main runs use H100 GPUs on Modal. The paraphrase task is the primary venue for scale and handoff analysis; sonnets stress-test preference construction under extreme low resource.

What They Find

Data scale beats epoch repetition

On Quora, moving from 2.8k to 283k examples cuts the train–dev loss gap and lifts dev F1 to 88.2. Under a fixed time budget, fewer epochs on more data beats many epochs on less data (e.g., 283k×2 epochs ≈ 86.2 F1 in ~15 minutes vs. 28.3k×10 epochs at 82.3 F1). For small models, diversity in supervision dominates memorization.

FFT beats LoRA; rank is not monotonic

At epoch 3 on the full dataset, FFT SFT reaches 89.24% accuracy vs. 86.34% for the best LoRA rank (r = 8). Surprisingly, r = 16 underperforms r = 8, which the authors attribute to under-trained high-rank adapters with the same update budget.

LoRA also does not reduce wall-clock on their H100 setup: training is compute-bound, not memory-bound, so saving trainable parameters does not translate into faster steps at GPT-2 scale.

DPO helps a little—and sometimes needs no warm start

After tuning (LR 5×10⁻⁶, β = 0.2), the best paraphrase pipeline is SFT@9 → DPO, reaching 90.05% dev accuracy vs. 89.87% for SFT alone—a gain under 0.2 points in accuracy spread across strategies (89.46–90.05).

DPO-only (no SFT warm start) still hits 89.46%, nearly matching SFT-only. When preferences are literally “right class vs. wrong class,” DPO can learn the same boundary as cross-entropy from scratch.

Staged handoff timing matters modestly: early SFT@3 → DPO underperforms a near-converged SFT@9 → DPO start, consistent with DPO as a light refinement stage.

Sonnets: SFT carries almost everything

On 131 training poems, DPO moves chrF by at most ~0.3 points over SFT. Augmenting to 325 pairs from only 65 prompts (V3) hurts—repeated prompts with different rejections overfit a narrow set. Temperature sweeps also push back on folklore: T = 0.5 is worst for chrF; T ≈ 1.5 works best with nucleus sampling at p = 0.9.

Qualitatively, SFT captures local Shakespearean diction but still drifts on meter, rhyme, and long-range argument—useful context for what preference data can and cannot fix at this scale.

Limitations

The study is intentionally narrow: one architecture family (GPT-2), one classification benchmark, one stylized generation task, single-seed runs for some comparisons, and H100-centric timing. Gains from LoRA or DPO may look different on memory-bound GPUs, larger models, or rich human preference data.

The authors are explicit that DPO differences are descriptive, not claimed as statistically significant given single runs.

Future Directions

Several extensions follow naturally from the paper’s conclusions and open questions.

Scale the backbone systematically

The central claim is regime-dependent. A logical next step is a capacity sweep (e.g., 124M → 1B → 7B) with matched data and compute budgets to map where DPO and LoRA start to dominate FFT—turning anecdotal “small model” advice into a curve.

Richer preferences for generation

Paraphrase DPO aligns with SFT because labels are discrete. Sonnet DPO uses synthetic rejections; human or LLM-judge preferences on style, coherence, and form might widen the gap between SFT and DPO when pairwise signal is not redundant with cross-entropy.

Statistical rigor and multi-seed reporting

Handoff and objective comparisons are within fractions of a point. Future work should report confidence intervals over seeds and data shards, especially for alignment stages where variance is easy to miss.

When LoRA is worth it

Here, LoRA saves little time on H100s. Studies on memory-limited devices, larger batch sizes enabled by PEFT, or multi-adapter serving would clarify where rank adaptation still wins despite lower accuracy.

Integration with modern alignment stacks

Extensions could add KTO, ORPO, or reward-model-based RL on the same small-model grid, asking whether newer objectives outperform DPO when data is scarce—or whether they collapse to the same margins as label-flip DPO.

Monitoring alignment stages in training dashboards

For practitioners, the lesson is to measure on the target regime before adopting a pipeline stage. Logging dev accuracy/F1 at SFT handoff, DPO β sweeps, and train–dev gaps—similar to multi-metric training monitors—would make “add DPO” a data-driven choice rather than a default.

Ideas for Our Work

The UF Data Studio often cares about whether a cheap intervention actually moves the metric that matters—fairness slices, subgroup F1, or conflict probes—not just aggregate loss.

Three parallels stand out.

Objective vs. infrastructure. On GPT-2, swapping SFT for DPO changes accuracy less than swapping LoRA for FFT. In our setting, a fancy alignment or PEFT step may matter less than full-data supervised training or better slice labels—echoing InsightBoard’s emphasis on seeing subgroup curves, not only headline accuracy.

Preference signal quality. DPO-only works when preferences duplicate supervision. That resembles evaluation design: if your “preference” benchmark does not stress anything beyond SFT, you should expect marginal gains. Conflict-style tests (prompt–image mismatch, entity-swap bias) exist precisely to measure behavior SFT alone does not constrain.

Low-resource honesty. Sonnet results warn against scaling preference pairs by repeating the same prompts. Small-data NLP projects in the lab face the same trap—augmentation that looks larger but narrows diversity can hurt. Reporting seed stability (as the authors do for chrF) is good practice we can adopt in course and research benchmarks.

For more information about our research, return to our homepage: ufdatastudio.com.