What is KL Divergence Constraint? Why does it exist? Where does it appear in Reinforcement Learning (RLHF)?
Draw or describe a visual model of KL Divergence Constraint. What does it look like conceptually?
Work through a concrete manual example of KL Divergence Constraint with real numbers or pseudocode.
How is KL Divergence Constraint different from related concepts in Reinforcement Learning (RLHF)? What are the trade-offs?
Explain KL Divergence Constraint in plain language using an analogy. Could a non-expert follow it?