RLHF
Reinforcement learning from human feedback (RLHF) aligns language models with human preferences by training a reward model on human rankings of model outputs, then optimizing the model against it through supervised fine-tuning, reward modeling, and reinforcement learning, a method that let a 1.3B-parameter InstructGPT be preferred by raters over the 175B-parameter GPT-3.[1][2][5] Its limits include costly preference data and open problems in feedback, reward models, and policy optimization, and alternatives such as Direct Preference Optimization (DPO) use a single objective on preference data without a separate reward model.[7][9][11]
Overview
The Hugging Face account of RLHF describes the process as pretraining a language model, collecting data to train a reward model, and fine-tuning the model with reinforcement learning.[2] The same account argues that a plain next-token loss does not capture subjective qualities such as creativity or truthfulness, which motivates the use of human preference data.[2]
RLHF was popularized by InstructGPT and is used to align assistants such as ChatGPT, Claude and Gemini.[10]
Pipeline
Sources name the stages differently, but they describe the same general sequence.[1][2]
Supervised fine-tuning
A pretrained model is fine-tuned on demonstrations of desired behavior. In InstructGPT, labeler-written demonstrations were used to fine-tune GPT-3.[5]
Reward model training
Human annotators rank model outputs. A reward model is trained on these rankings; it takes text and outputs a scalar score intended to reflect human preference.[2]
Reinforcement learning
The reward model guides a reinforcement learning step that updates the language model.[5] The InstructGPT authors discuss an “alignment tax”, a performance cost that can be reduced by mixing pretraining data into the reinforcement learning step.[6]
History
Learning from human preferences
Christiano et al. first posted “Deep Reinforcement Learning from Human Preferences” on 2017-06-12, and it was published at NeurIPS 2017. The authors are Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei.[3] The method learns a reward function from non-expert human comparisons of pairs of trajectory segments, then trains the agent on that learned reward.[3]
The authors report that the approach solved Atari and simulated robot locomotion tasks without access to the true reward, using feedback on less than 1% of the agent’s interactions.[3] OpenAI’s companion post describes a backflip agent trained with about 900 bits of feedback, and notes that performance is limited by the evaluator’s intuition about correct behavior.[4]
Application to language models
Ouyang et al. of OpenAI applied the approach to instruction following in “Training language models to follow instructions with human feedback”, presented at NeurIPS 2022.[5][12] Human raters preferred outputs from the 1.3B-parameter InstructGPT over those of the 175B-parameter GPT-3. The authors also report improved truthfulness, reduced toxic output, and minimal regressions on public NLP datasets, while noting that the models still make simple mistakes.[5]
Limitations
Casper et al. survey open problems with RLHF, published on arXiv in 2023 and later in TMLR. They describe RLHF as the main method for fine-tuning large language models, with little prior public work systematizing its flaws.[7] The survey groups challenges into feedback, reward model and policy optimization issues, distinguishes tractable challenges from fundamental limitations, and proposes auditing and disclosure standards.[8]
Sourcing high-quality preference data is also described as expensive.[11]
Alternatives
Rafailov et al. introduced Direct Preference Optimization (DPO) at NeurIPS 2023. They characterize the standard RLHF pipeline as complex and unstable. DPO uses a single classification-style objective on preference data, and needs no separate reward model, no sampling during fine-tuning, and little hyperparameter tuning.[9] The authors report that DPO matches or beats existing methods, exceeds RLHF at controlling sentiment, and improves summarization and single-turn dialogue.[9]
Sources differ on scope. Rafailov et al. present DPO as replacing the RLHF pipeline,[9] whereas the Casper et al. survey addresses RLHF “and related methods”.[8] Whether DPO is classified as a form of RLHF is therefore a matter of framing.
See also
- Human preference data collection
- AI data labs