Describes the InstructGPT pipeline — supervised fine-tuning, a learned reward model and reinforcement learning — and reports that labellers preferred a much smaller fine-tuned model to the original large model.
This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
This is the clearest public description of how instruction-following assistants were first trained at scale. Its limitations section is as instructive as its results: it is explicit about whose preferences the model ends up reflecting.
What you will learn
- Describe the three stages of an RLHF pipeline
- Discuss what “alignment” means when it is measured by labeller preference
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Deep Reinforcement Learning from Human Preferences (recommended)
How to cite
Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems 35.