Implement a DPO Training Step

Your team wants to align a language model to human preferences without the complexity of RLHF. DPO reformulates the preference learning problem as a simple classification loss over pairs of (chosen, rejected) responses. Given a policy model and a frozen reference model, DPO adjusts the policy so that chosen responses become more likely relative to rejected ones, while staying close to the reference. You're given a SimpleLanguageModel with embedding and output projection. Implement the sequence log probability computation, the DPO loss formula, and a gradient step that updates only the policy model's parameters. The reference model must remain frozen throughout training.

1. 1.Implement log_prob_of_sequence using log_softmax and teacher-forced next-token prediction
2. 2.Implement dpo_loss using the standard DPO formula with policy and reference log probs
3. 3.Apply the sigmoid/log-sigmoid transform to the reward difference scaled by beta
4. 4.Implement a numerical gradient step that perturbs policy parameters
5. 5.Ensure the reference model parameters are never modified during training