RLHF

Reinforcement learning from human feedback (RLHF) aligns language models with human preferences by training a reward model on human rankings of model outputs, then optimizing the model against it through supervised fine-tuning, reward modeling, and reinforcement learning, a method that let a 1.3B-parameter InstructGPT be preferred by raters over the 175B-parameter GPT-3.[1][2][5] Its limits include costly preference data and open problems in feedback, reward models, and policy optimization, and alternatives such as Direct Preference Optimization (DPO) use a single objective on preference data without a separate reward model.[7][9][11]

Overview

The Hugging Face account of RLHF describes the process as pretraining a language model, collecting data to train a reward model, and fine-tuning the model with reinforcement learning.[2] The same account argues that a plain next-token loss does not capture subjective qualities such as creativity or truthfulness, which motivates the use of human preference data.[2]

RLHF was popularized by InstructGPT and is used to align assistants such as ChatGPT, Claude and Gemini.[10]

Pipeline

Sources name the stages differently, but they describe the same general sequence.[1][2]

Supervised fine-tuning

A pretrained model is fine-tuned on demonstrations of desired behavior. In InstructGPT, labeler-written demonstrations were used to fine-tune GPT-3.[5]

Reward model training

Human annotators rank model outputs. A reward model is trained on these rankings; it takes text and outputs a scalar score intended to reflect human preference.[2]

Reinforcement learning

The reward model guides a reinforcement learning step that updates the language model.[5] The InstructGPT authors discuss an “alignment tax”, a performance cost that can be reduced by mixing pretraining data into the reinforcement learning step.[6]

History

Learning from human preferences

Christiano et al. first posted “Deep Reinforcement Learning from Human Preferences” on 2017-06-12, and it was published at NeurIPS 2017. The authors are Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei.[3] The method learns a reward function from non-expert human comparisons of pairs of trajectory segments, then trains the agent on that learned reward.[3]

The authors report that the approach solved Atari and simulated robot locomotion tasks without access to the true reward, using feedback on less than 1% of the agent’s interactions.[3] OpenAI’s companion post describes a backflip agent trained with about 900 bits of feedback, and notes that performance is limited by the evaluator’s intuition about correct behavior.[4]

Application to language models

Ouyang et al. of OpenAI applied the approach to instruction following in “Training language models to follow instructions with human feedback”, presented at NeurIPS 2022.[5][12] Human raters preferred outputs from the 1.3B-parameter InstructGPT over those of the 175B-parameter GPT-3. The authors also report improved truthfulness, reduced toxic output, and minimal regressions on public NLP datasets, while noting that the models still make simple mistakes.[5]

Limitations

Casper et al. survey open problems with RLHF, published on arXiv in 2023 and later in TMLR. They describe RLHF as the main method for fine-tuning large language models, with little prior public work systematizing its flaws.[7] The survey groups challenges into feedback, reward model and policy optimization issues, distinguishes tractable challenges from fundamental limitations, and proposes auditing and disclosure standards.[8]

Sourcing high-quality preference data is also described as expensive.[11]

Alternatives

Rafailov et al. introduced Direct Preference Optimization (DPO) at NeurIPS 2023. They characterize the standard RLHF pipeline as complex and unstable. DPO uses a single classification-style objective on preference data, and needs no separate reward model, no sampling during fine-tuning, and little hyperparameter tuning.[9] The authors report that DPO matches or beats existing methods, exceeds RLHF at controlling sentiment, and improves summarization and single-turn dialogue.[9]

Sources differ on scope. Rafailov et al. present DPO as replacing the RLHF pipeline,[9] whereas the Casper et al. survey addresses RLHF “and related methods”.[8] Whether DPO is classified as a form of RLHF is therefore a matter of framing.

See also

  • Human preference data collection
  • AI data labs

References

  1. Snowflake — Reinforcement learning from human feedback (RLHF). https://www.snowflake.com/en/artificial-intelligence/machine-learning/reinforcement-learning/rlhf/
  2. Hugging Face — Illustrating Reinforcement Learning from Human Feedback (RLHF). https://huggingface.co/blog/rlhf
  3. Christiano et al. — Deep Reinforcement Learning from Human Preferences (arXiv 1706.03741). https://arxiv.org/abs/1706.03741v3
  4. OpenAI — Learning from human preferences. https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/
  5. Ouyang et al. — Training language models to follow instructions with human feedback (arXiv 2203.02155). https://huggingface.co/papers/2203.02155
  6. Ouyang et al. — Training language models to follow instructions with human feedback (full text). https://ar5iv.labs.arxiv.org/html/2203.02155
  7. Casper et al. — Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (arXiv 2307.15217). https://huggingface.co/papers/2307.15217
  8. Casper et al. — Open Problems and Fundamental Limitations of RLHF (ETH Zurich Research Collection). https://research-collection.ethz.ch/handle/20.500.11850/651806
  9. Rafailov et al. — Direct Preference Optimization (NeurIPS 2023). https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract.html
  10. Real Python — Reinforcement learning from human feedback. https://realpython.com/ref/ai-coding-glossary/reinforcement-learning-from-human-feedback/
  11. Wikiwand — Reinforcement learning from human feedback. https://wikiwand.com/en/Reinforcement_learning_from_human_feedback
  12. NeurIPS 2022 — Training language models to follow instructions with human feedback (poster). https://neurips.cc/virtual/2022/poster/52886

This page was last edited on (UTC).