SkillCompiler/data/skills-bench/tasks/simpo-code-reproduction/environment/SimPO/paper.json

{
  "1": "SimPO: Simple Preference Optimization\nwith a Reference-Free Reward\nYu Meng1∗\nMengzhou Xia2 ∗\nDanqi Chen2\n1Computer Science Department, University of Virginia\n2Princeton Language and Intelligence (PLI), Princeton University\nyumeng5@virginia.edu\n{mengzhou,danqic}@cs.princeton.edu\nAbstract\nDirect Preference Optimization (DPO) is a widely used offline preference opti-\nmization algorithm that reparameterizes reward functions in reinforcement learning\nfrom human feedback (RLHF) to enhance simplicity and training stability. In this\nwork, we propose SimPO, a simpler yet more effective approach. The effectiveness\nof SimPO is attributed to a key design: using the average log probability of a\nsequence as the implicit reward. This reward formulation better aligns with model\ngeneration and eliminates the need for a reference model, making it more compute\nand memory efficient. Additionally, we introduce a target reward margin to the\nBradley-Terry objective to encourage a larger margin between the winning and\nlosing responses, further improving the algorithm’s performance. We compare\nSimPO to DPO and its recent variants across various state-of-the-art training setups,\nincluding both base and instruction-tuned models such as Mistral, Llama 3, and\nGemma 2. We evaluate on extensive chat-based evaluation benchmarks, including\nAlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO\nconsistently and significantly outperforms existing approaches without substantially\nincreasing response length. Specifically, SimPO outperforms DPO by up to 6.4\npoints on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing\nmodel, built on Gemma-2-9B-it, achieves a 72.4% length-controlled win rate on\nAlpacaEval 2, a 59.1% win rate on Arena-Hard, and ranks 1st on Chatbot Arena\namong <10B models with real user votes.1\n1\nIntroduction\nLearning from human feedback is crucial in aligning large language models (LLMs) with human\nvalues and intentions [51], ensuring they are helpful, honest, and harmless [5]. Reinforcement learning\nfrom human feedback (RLHF) [18, 62, 73] is a popular method for fine-tuning language models\nto achieve effective alignment. While the classical RLHF approach [62, 70] has shown impressive\nresults, it presents optimization challenges due to its multi-stage procedure, which involves training a\nreward model and then optimizing a policy model to maximize that reward [13].\nRecently, researchers have been exploring simpler offline algorithms. Direct Preference Optimization\n(DPO) [66] is one such approach. DPO reparameterizes the reward function in RLHF to directly\nlearn a policy model from preference data, eliminating the need for an explicit reward model. It has\ngained widespread practical adoption due to its simplicity and stability. In DPO, the implicit reward\nis formulated using the log ratio of the likelihood of a response between the current policy model and\nthe supervised fine-tuned (SFT) model. However, this reward formulation is not directly aligned with\n∗Equal Contribution.\n1Code and models can be found at https://github.com/princeton-nlp/SimPO.\n38th Conference on Neural Information Processing Systems (NeurIPS 2024).\narXiv:2405.14734v3  [cs.CL]  1 Nov 2024\n",
  "2": "0\n25\n50\nAlpacaEval 2 LC Win Rate (%)\nLlama3
\nBase 8B\nLlama3
\nInstruct 8B\nMistral
\nBase 7B\nMistral
\nInstruct 7B\nLlama3
\nBase 8B\nLlama3
\nInstruct 8B\nMistral
\nBase 7B\nMistral
\nInstruct 7B\nArena-Hard Win Rate (%)\n0\n20\n40\n+3.8\n+4.4\n+6.4\n+5.3\n+7.5\n+1.2\n+6.2\n+4.5\nDPO\nSimPO\nDPO\nSimPO\nFigure 1: SimPO and DPO mainly differ in their reward formulation, as indicated in the shaded box.\nSimPO outperforms DPO significantly across a range of settings on AlpacaEval 2 and Arena-Hard.\nthe metric used to guide generation, which is approximately the average log likelihood of a response\ngenerated by the policy model. We hypothesize that this discrepancy between training and inference\nmay lead to suboptimal performance.\nTable 1: Length-controlled (LC) and raw win rate (WR),\nand generation lengths of top models on the AlpacaEval\n2 Leaderboard. Bold are the models we trained.\nModel\nLC (%) WR (%) Len.\nGemma-2-9B-it-SimPO\n72.4\n65.9\n1833\nGPT-4 Turbo (04/09)\n55.0\n46.1\n1802\nGemma-2-9B-it\n51.1\n38.1\n1571\nLlama-3-8B-Instruct-SimPO\n44.7\n40.5\n1825\nClaude 3 Opus\n40.5\n29.1\n1388\nLlama-3-8B-Instruct-DPO\n40.3\n37.9\n1837\nLlama-3-70B-Instruct\n34.4\n33.2\n1919\nLlama-3-8B-Instruct\n26.0\n25.3\n1899\nIn this work, we propose SimPO, a simple\nyet effective offline preference optimiza-\ntion algorithm (Figure 1).\nThe core of\nour algorithm aligns the reward function\nin the preference optimization objective\nwith the generation metric. SimPO consists\nof two major components: (1) a length-\nnormalized reward, calculated as the aver-\nage log probability of all tokens in a re-\nsponse using the policy model, and (2) a\ntarget reward margin to ensure the reward\ndifference between winning and losing re-\nsponses exceeds this margin. In summary,\nSimPO has the following properties:\n• Simplicity: SimPO does not require a reference model, making it more lightweight and easier to\nimplement compared to DPO and other reference-based methods.\n• Significant performance advantage: Despite its simplicity, SimPO significantly outperforms\nDPO and its latest variants (e.g., a recent reference-free objective ORPO [42]). The performance\nadvantage is consistent across various training setups and extensive chat-based evaluations, includ-\ning AlpacaEval 2 [55, 28] and the challenging Arena-Hard [54] benchmark. It achieves up to a\n6.4 point improvement on AlpacaEval 2 and a 7.5 point improvement on Arena-Hard compared to\nDPO (Figure 1).\n• Minimal length exploitation: SimPO does not significantly increase response length compared to\nthe SFT or DPO models (Table 1), indicating minimal length exploitation [28, 71, 85].\nExtensive analysis shows that SimPO utilizes preference data more effectively, leading to a more\naccurate likelihood ranking of winning and losing responses on a held-out validation set, which in\nturn translates to a better policy model. As shown in Table 1, our Gemma-2-9B-it-SimPO model\nachieves state-of-the-art performance, with a 72.4% length-controlled win rate on AlpacaEval 2\nand a 59.1% win rate on Arena-Hard, establishing it as the strongest open-source model under 10B\nparameters. Most notably, when evaluated on Chatbot Arena [17] with real user votes, our model\nsignificantly improved upon the initial Gemma-2-9B-it model, advancing from 36th to 25th place and\nranking first among all <10B models on the leaderboard.2\n2\nSimPO: Simple Preference Optimization\nIn this section, we first introduce the background of DPO (§2.1). Then we identify the discrepancy\nbetween DPO’s reward and the likelihood metric used for generation, and propose an alternative\nreference-free reward formulation that mitigates this issue (§2.2). Finally, we derive the SimPO\nobjective by incorporating a target reward margin term into the Bradley-Terry model (§2.3).\n2As of September 16th, 2024.\n2\n",
  "3": "2.1\nBackground: Direct Preference Optimization (DPO)\nDPO [66] is one of the most popular preference optimization methods. Instead of learning an explicit\nreward model [62], DPO reparameterizes the reward function r using a closed-form expression with\nthe optimal policy:\nr(x, y) = β log πθ(y | x)\nπref(y | x) + β log Z(x),\n(1)\nwhere πθ is the policy model, πref is the reference policy, typically the supervised fine-tuned (SFT)\nmodel, and Z(x) is the partition function. By incorporating this reward formulation into the Bradley-\nTerry (BT) ranking objective [11], p(yw ≻yl | x) = σ (r(x, yw) −r(x, yl)), DPO expresses the\nprobability of preference data with the policy model rather than the reward model, yielding the\nfollowing objective:\nLDPO(πθ; πref) = −E(x,yw,yl)∼D\n\u0014\nlog σ\n\u0012\nβ log πθ(yw | x)\nπref(yw | x) −β log πθ(yl | x)\nπref(yl | x)\n\u0013\u0015\n,\n(2)\nwhere (x, yw, yl) are preference pairs consisting of the prompt, the winning response, and the losing\nresponse from the preference dataset D.\n2.2\nA Simple Reference-Free Reward Aligned with Generation\nDiscrepancy between reward and generation for DPO.\nUsing Eq. (1) as the implicit reward has\nthe following drawbacks: (1) it requires a reference model πref during training, which incurs additional\nmemory and computational costs; and (2) it creates a mismatch between the reward optimized in\ntraining and the log-likelihood optimized during inference, where no reference model is involved.\nThis means that in DPO, for any triple (x, yw, yl), satisfying the reward ranking r(x, yw) > r(x, yl)\ndoes not necessarily mean that the likelihood ranking pθ(yw | x) > pθ(yl | x) is met (here pθ is the\naverage log-likelihood in Eq. (3)). In our experiments, we observed that only ∼50% of the triples\nfrom the training set satisfy this condition when trained with DPO (Figure 4b). This observation\naligns with a concurrent work [14], which finds that existing models trained with DPO exhibit random\nranking accuracy in terms of average log-likelihood, even after extensive preference optimization.\nLength-normalized reward formulation.\nOne solution is to use the summed token log probability\nas the reward, but this suffers from length bias–longer sequences tend to have lower log probabilities.\nConsequently, when yw is longer than yl, optimizing the summed log probability as a reward forces\nthe model to artificially inflate probabilities for longer sequences to ensure yw receives a higher\nreward than yl. This overcompensation increases the risk of degeneration. To address this issue, we\nconsider using the average log-likelihood as the implicit reward:\npθ(y | x) = 1\n|y| log πθ(y | x) = 1\n|y|\n|y|\nX\ni=1\nlog πθ(yi | x, y<i).\n(3)\nThis metric is commonly used for ranking options in beam search [35, 53] and multiple-choice tasks\nwithin language models [12, 41, 62]. Naturally, we consider replacing the reward formulation in DPO\nwith pθ in Eq. (3), so that it aligns with the likelihood metric that guides generation. This results in a\nlength-normalized reward:\nrSimPO(x, y) = β\n|y| log πθ(y | x) = β\n|y|\n|y|\nX\ni=1\nlog πθ(yi | x, y<i),\n(4)\nwhere β is a constant that controls the scaling of the reward difference. We find that normalizing the\nreward with response lengths is crucial; removing the length normalization term from the reward\nformulation results in a bias toward generating longer but lower-quality sequences (see Section 4.4\nfor more details). Consequently, this reward formulation eliminates the need for a reference model,\nenhancing memory and computational efficiency compared to reference-dependent algorithms.\n2.3\nThe SimPO Objective\nTarget reward margin.\nAdditionally, we introduce a target reward margin term, γ > 0, to the\nBradley-Terry objective to ensure that the reward for the winning response, r(x, yw), exceeds the\n3\n",
  "4": "reward for the losing response, r(x, yl), by at least γ:\np(yw ≻yl | x) = σ (r(x, yw) −r(x, yl) −γ) .\n(5)\nThe margin between two classes is known to influence the generalization capabilities of classifiers [1,\n10, 22, 31].3 In standard training settings with random model initialization, increasing the target\nmargin typically improves generalization. In preference optimization, the two classes are the winning\nand losing responses for a single input. In practice, we observe that generation quality initially\nimproves with an increasing target margin but degrades when the margin becomes too large (§4.3).\nOne of DPO’s variants, IPO [6], also formulates a target reward margin similar to SimPO. However,\nits full objective is not as effective as SimPO (§4.1).\nObjective.\nFinally, we obtain the SimPO objective by plugging Eq. (4) into Eq. (5):\nLSimPO(πθ) = −E(x,yw,yl)∼D\n\u0014\nlog σ\n\u0012 β\n|yw| log πθ(yw|x) −β\n|yl| log πθ(yl|x) −γ\n\u0013\u0015\n.\n(6)\nIn summary, SimPO employs an implicit reward formulation that directly aligns with the generation\nmetric, eliminating the need for a reference model. Additionally, it introduces a target reward margin\nγ to help separating the winning and losing responses. In Appendix F, we provide a gradient analysis\nof SimPO and DPO to further understand the differences between the two methods.\nPreventing catastrophic forgetting without KL regularization.\nAlthough SimPO does not impose\nKL regularization, we find that a combination of practical factors ensures effective learning from\npreference data while maintaining generalization, leading to an empirically low KL divergence from\nthe reference model. These factors are: (1) a small learning rate, (2) a preference dataset that covers\ndiverse domains and tasks, and (3) the intrinsic robustness of LLMs to learn from new data without\nforgetting prior knowledge. We present KL divergence experiments in Section 4.4.\n3\nExperimental Setup\nModels and training settings.\nWe perform preference optimization with two families of models,\nLlama-3-8B [2] and Mistral-7B [44], under two setups: Base and Instruct. In this section, our goal\nis to understand the performance of SimPO vs. other preference optimization methods in different\nexperimental setups. Our strongest model is based on Gemma-2-9B (Instruct setup) with a stronger\nreward model, RLHFlow/ArmoRM-Llama3-8B-v0.1 [84] (Table 1). We will present and discuss\nthese results in Appendix J.\nFor the Base setup, we follow the training pipeline of Zephyr [80]. First, we train a base model (i.e.,\nmistralai/Mistral-7B-v0.1, or meta-llama/Meta-Llama-3-8B) on the UltraChat-200k dataset [25] to\nobtain an SFT model. Then, we perform preference optimization on the UltraFeedback dataset [23]\nusing the SFT model as the starting point. This setup provides a high level of transparency, as the\nSFT models are trained on open-source data.\nFor the Instruct setup, we use an off-the-shelf instruction-tuned model (i.e., meta-llama/Meta-\nLlama-3-8B-Instruct, or mistralai/Mistral-7B-Instruct-v0.2) as the SFT models.4 These models have\nundergone extensive instruction-tuning processes, making them more powerful and robust than the\nSFT models in the Base setup. However, they are also more opaque because their RLHF procedure\nis not publicly disclosed. To mitigate the distribution shift between SFT models and the preference\noptimization process, we generate the preference dataset using the SFT models following [79].\nThis makes our Instruct setup closer to an on-policy setting. Specifically, we use prompts from\nthe UltraFeedback dataset and regenerate the chosen and rejected response pairs (yw, yl) with the\nSFT models. For each prompt x, we generate 5 responses using the SFT model with a sampling\ntemperature of 0.8. We then use llm-blender/PairRM [45] to score the 5 responses, selecting the\n3This margin is termed home advantage in Bradley-Terry models [1, 31].\n4It is unclear whether the released instruct checkpoints have undergone supervised fine-tuning (SFT) or the\ncomplete RLHF pipeline. For simplicity, we refer to these checkpoints as SFT models.\n4\n",
  "5": "Table 2: Evaluation details for AlpacaEval 2 [55], Arena-Hard [54], and MT-Bench [99]. The baseline\nmodel refers to the model compared against. GPT-4 Turbo corresponds to GPT-4-Preview-1106.\n# Exs. Baseline Model\nJudge Model\nScoring Type\nMetric\nAlpacaEval 2\n805\nGPT-4 Turbo\nGPT-4 Turbo\nPairwise comparison\nLC & raw win rate\nArena-Hard\n500\nGPT-4-0314\nGPT-4 Turbo\nPairwise comparison\nWin rate\nMT-Bench\n80\n-\nGPT-4/GPT-4 Turbo Single-answer grading\nRating of 1-10\nhighest-scoring one as yw and the lowest-scoring one as yl. We only generated data in a single pass\ninstead of iteratively as in [79].5\nIn summary, we have four setups: Llama-3-Base, Llama-3-Instruct, Mistral-Base, and Mistral-Instruct.\nWe believe these configurations represent the state-of-the-art, placing our models among the top\nperformers on various leaderboards. We encourage future research to adopt these settings for better\nand fairer comparisons of different algorithms. Additionally, we find that tuning hyperparameters is\ncrucial for achieving optimal performance with all the offline preference optimization algorithms,\nincluding DPO and SimPO. Generally, for SimPO, setting β between 2.0 and 2.5 and γ between 0.5\nand 1.5 leads to good performance across all setups. For more details, please refer to Appendix B.\nEvaluation benchmarks.\nWe primarily assess our models using three of the most popular open-\nended instruction-following benchmarks: MT-Bench [99], AlpacaEval 2 [55], and Arena-Hard\nv0.1 [54]. These benchmarks evaluate the models’ versatile conversational abilities across a diverse\nset of queries and have been widely adopted by the community (details in Table 2). AlpacaEval 2\nconsists of 805 questions from 5 datasets, and MT-Bench covers 8 categories with 80 questions. The\nmost recently released Arena-Hard is an enhanced version of an MT-Bench, incorporating 500 well-\ndefined technical problem-solving queries. We report scores following each benchmark’s evaluation\nprotocol. For AlpacaEval 2, we report both the raw win rate (WR) and the length-controlled win\nrate (LC) [28]. The LC metric is specifically designed to be robust against model verbosity. For\nArena-Hard, we report the win rate (WR) against the baseline model. For MT-Bench, we report the\naverage MT-Bench score with GPT-4 and GPT-4-Preview-1106 as the judge model.6 For decoding\ndetails, please refer to Appendix B. We also evaluate on downstream tasks from the Huggingface\nOpen Leaderboard benchmarks [9], with additional details in in Appendix C.\nTable 3: Various preference optimization objectives given pref-\nerence data D = (x, yw, yl), where x is an input, and yw and yl\nare the winning and losing responses.\nMethod\nObjective\nRRHF [91]\nmax\n\u0010\n0, −\n1\n|yw| log πθ(yw|x) +\n1\n|yl| log πθ(yl|x)\n\u0011\n−λ log πθ(yw|x)\nSLiC-HF [96] max (0, δ −log πθ(yw|x) + log πθ(yl|x)) −λ log πθ(yw|x)\nDPO [66]\n−log σ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −β log πθ(yl|x)\nπref(yl|x)\n\u0011\nIPO [6]\n\u0010\nlog πθ(yw|x)\nπref(yw|x) −log πθ(yl|x)\nπref(yl|x) −\n1\n2τ\n\u00112\nCPO [88]\n−log σ (β log πθ(yw|x) −β log πθ(yl|x)) −λ log πθ(yw|x)\nKTO [29]\n−λwσ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −zref\n\u0011\n+ λlσ\n\u0010\nzref −β log πθ(yl|x)\nπref(yl|x)\n\u0011\n,\nwhere zref = E(x,y)∼D [βKL (πθ(y|x)||πref(y|x))]\nORPO [42]\n−log pθ(yw|x) −λ log σ\n\u0010\nlog\npθ(yw|x)\n1−pθ(yw|x) −log\npθ(yl|x)\n1−pθ(yl|x)\n\u0011\n,\nwhere pθ(y|x) = exp\n\u0010\n1\n|y| log πθ(y|x)\n\u0011\nR-DPO [64]\n−log σ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −β log πθ(yl|x)\nπref(yl|x) + (α|yw| −α|yl|)\n\u0011\nSimPO\n−log σ\n\u0010\nβ\n|yw| log πθ(yw|x) −\nβ\n|yl| log πθ(yl|x) −γ\n\u0011\nBaselines.\nWe compare SimPO\nwith\nother\noffline\npreference\noptimization\nmethods\nlisted\nin Table 3.7\nRRHF [91] and\nSLiC-HF [96] are ranking losses.\nRRHF\nuses\nlength-normalized\nlog-likelihood, similar to SimPO’s\nreward\nfunction,\nwhile\nSLiC-\nHF uses log-likelihood directly\nand includes an SFT objective.\nIPO [6] is a theoretically grounded\napproach\nmethod\nthat\navoids\nDPO’s assumption that pairwise\npreferences can be replaced with\npointwise rewards.\nCPO [88]\nuses sequence likelihood as a\nreward and trains alongside an\nSFT objective. KTO [29] learns\nfrom non-paired preference data.\n5We also experimented with using a stronger reward model, RLHFlow/ArmoRM-Llama3-8B-v0.1 [84], to\nrank generated data, which yields significantly improved performance (see Appendix H and Appendix J). This is\nthe reward model we used in our Gemma 2 experiments.\n6GPT-4-Preview-1106 produces more accurate reference answers and judgments compared to GPT-4.\n7Many recent studies [90, 74] have extensively compared DPO and PPO [70]. We will leave the comparison\nof PPO and SimPO to future work.\n5\n",
  "6": "Table 4: AlpacaEval 2 [55], Arena-Hard [54], and MT-Bench [99] results under the four settings.\nLC and WR denote length-controlled and raw win rate, respectively. We train SFT models for Base\nsettings on the UltraChat dataset. For Instruct settings, we use off-the-shelf models as the SFT model.\nMethod\nMistral-Base (7B)\nMistral-Instruct (7B)\nAlpacaEval 2 Arena-Hard\nMT-Bench\nAlpacaEval 2 Arena-Hard\nMT-Bench\nLC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4 LC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4\nSFT\n8.4\n6.2\n1.3\n4.8\n6.3\n17.1\n14.7\n12.6\n6.2\n7.5\nRRHF [91]\n11.6\n10.2\n5.8\n5.4\n6.7\n25.3\n24.8\n18.1\n6.5\n7.6\nSLiC-HF [96]\n10.9\n8.9\n7.3\n5.8\n7.4\n24.1\n24.6\n18.9\n6.5\n7.8\nDPO [66]\n15.1\n12.5\n10.4\n5.9\n7.3\n26.8\n24.9\n16.3\n6.3\n7.6\nIPO [6]\n11.8\n9.4\n7.5\n5.5\n7.2\n20.3\n20.3\n16.2\n6.4\n7.8\nCPO [88]\n9.8\n8.9\n6.9\n5.4\n6.8\n23.8\n28.8\n22.6\n6.3\n7.5\nKTO [29]\n13.1\n9.1\n5.6\n5.4\n7.0\n24.5\n23.6\n17.9\n6.4\n7.7\nORPO [42]\n14.7\n12.2\n7.0\n5.8\n7.3\n24.5\n24.9\n20.8\n6.4\n7.7\nR-DPO [64]\n17.4\n12.8\n8.0\n5.9\n7.4\n27.3\n24.5\n16.1\n6.2\n7.5\nSimPO\n21.5\n20.8\n16.6\n6.0\n7.3\n32.1\n34.8\n21.0\n6.6\n7.6\nMethod\nLlama-3-Base (8B)\nLlama-3-Instruct (8B)\nAlpacaEval 2 Arena-Hard\nMT-Bench\nAlpacaEval 2 Arena-Hard\nMT-Bench\nLC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4 LC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4\nSFT\n6.2\n4.6\n3.3\n5.2\n6.6\n26.0\n25.3\n22.3\n6.9\n8.1\nRRHF [91]\n12.1\n10.1\n6.3\n5.8\n7.0\n31.3\n28.4\n26.5\n6.7\n7.9\nSLiC-HF [96]\n12.3\n13.7\n6.0\n6.3\n7.6\n26.9\n27.5\n26.2\n6.8\n8.1\nDPO [66]\n18.2\n15.5\n15.9\n6.5\n7.7\n40.3\n37.9\n32.6\n7.0\n8.0\nIPO [6]\n14.4\n14.2\n17.8\n6.5\n7.4\n35.6\n35.6\n30.5\n7.0\n8.3\nCPO [88]\n10.8\n8.1\n5.8\n6.0\n7.4\n28.9\n32.2\n28.8\n7.0\n8.0\nKTO [29]\n14.2\n12.4\n12.5\n6.3\n7.8\n33.1\n31.8\n26.4\n6.9\n8.2\nORPO [42]\n12.2\n10.6\n10.8\n6.1\n7.6\n28.5\n27.4\n25.8\n6.8\n8.0\nR-DPO [64]\n17.6\n14.4\n17.2\n6.6\n7.5\n41.1\n37.8\n33.1\n7.0\n8.0\nSimPO\n22.0\n20.3\n23.4\n6.6\n7.7\n44.7\n40.5\n33.8\n7.0\n8.0\nORPO [42]8 introduces a reference-model-free odd ratio term to directly contrast winning and losing\nresponses with the policy model and jointly trains with the SFT objective. R-DPO [64] is a modified\nversion of DPO that includes an additional regularization term to prevent exploitation of length. We\nthoroughly tune the hyperparameters for each baseline and report the best performance. We find that\nmany variants of DPO do not empirically present an advantage over standard DPO. Further details\ncan be found in Appendix B.\n4\nExperimental Results\nIn this section, we present main results of our experiments, highlighting the superior performance of\nSimPO on various benchmarks and ablation studies (§4.1). We provide an in-depth understanding\nof the following components: (1) length normalization (§4.2), (2) the margin term γ (§4.3), and (3)\nwhy SimPO outperforms DPO (§4.4). Unless otherwise specified, the ablation studies are conducted\nusing the Mistral-Base setting.\n4.1\nMain Results and Ablations\nSimPO consistently and significantly outperforms existing preference optimization methods.\nAs shown in Table 4, while all preference optimization algorithms enhance performance over the SFT\nmodel, SimPO, despite its simplicity, achieves the best overall performance across all benchmarks and\nsettings. These consistent and significant improvements highlight the robustness and effectiveness of\nSimPO. Notably, SimPO outperforms the best baseline by 3.6 to 4.8 points on the AlpacaEval 2 LC\nwin rate across various settings. On Arena-Hard, SimPO consistently achieves superior performance,\n8ORPO can directly train on preference data without the SFT stage. For fair comparisons, we start ORPO\nfrom the same SFT checkpoints as other baselines, which yields better results than starting from base checkpoints.\n6\n",
  "7": "Table 5: Ablation studies under Mistral-Base and Mistral-Instruct settings. We ablate each key design\nof SimPO: (1) removing length normalization in Eq. (4) (i.e., w/o LN); (2) setting target reward\nmargin γ to be 0 in Eq. (6) (i.e., γ = 0).\nMethod\nMistral-Base (7B) Setting\nMistral-Instruct (7B) Setting\nAlpacaEval 2 Arena-Hard\nMT-Bench\nAlpacaEval 2 Arena-Hard\nMT-Bench\nLC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4 LC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4\nDPO\n15.1\n12.5\n10.4\n5.9\n7.3\n26.8\n24.9\n16.3\n6.3\n7.6\nSimPO\n21.5\n20.8\n16.6\n6.0\n7.3\n32.1\n34.8\n21.0\n6.6\n7.6\nw/o LN\n11.9\n13.2\n9.4\n5.5\n7.3\n19.1\n19.7\n16.3\n6.4\n7.6\nγ = 0\n16.8\n14.3\n11.7\n5.6\n6.9\n30.9\n34.2\n20.5\n6.6\n7.7\n−200\n−100\n0\n100\n200\n300\nLength diff. |yw| −|yl|\n−10\n0\n10\nr(x, yw) −r(x, yl)\nπSimPO\nπSimPO w/o LN\nπSFT\n(a) Reward optimization.\n0\n200\n400\n600\n800\n1000\nResponse length |y|\n−15\n−10\n−5\n0\nAvg. log prob. pθ(y|x)\nρ = 0.34\n(b) SimPO.\n0\n200\n400\n600\n800\n1000\nResponse length |y|\n−15\n−10\n−5\n0\nAvg. log prob. pθ(y|x)\nρ = 0.82\n(c) SimPO without LN.\nFigure 2: Effect of length normalization (LN). (a) Relationship between reward margin and length\ndifference between winning and losing responses. (b) Spearman correlation between average log\nprobability and response length for SimPO. (c) Spearman correlation for SimPO without LN.\nthough it is occasionally surpassed by CPO [88]. We find that CPO generates responses that are, on\naverage, 50% longer than those generated by SimPO (See Table 10). Arena-Hard might favor longer\ngenerations due to the absence of a length penalty in its evaluation.\nBenchmark quality varies.\nAlthough all three benchmarks are widely adopted, we find that MT-\nBench exhibits poor separability across different methods. Minor differences between methods on\nMT-Bench may be attributed to randomness, likely due to the limited scale of its evaluation data and\nits single-instance scoring protocol. This finding aligns with observations reported in [54]. In contrast,\nAlpacaEval 2 and Arena-Hard provide more meaningful distinctions between different methods. We\nobserve that the win rate on Arena-Hard is significantly lower than on AlpacaEval 2, indicating that\nArena-Hard is a more challenging benchmark.9\nThe Instruct setting introduces significant performance gains.\nAcross all benchmarks, we\nobserve that the Instruct setting consistently outperforms the Base setting. This improvement is likely\ndue to the higher quality of SFT models used for initialization and the generation of more high-quality\npreference data by these models.\nBoth key designs in SimPO are crucial.\nIn Table 5, we demonstrate results from ablating each\nkey design of SimPO: (1) removing length normalization in Eq. (4) (i.e., w/o LN); (2) setting the\ntarget reward margin to be 0 in Eq. (6) (i.e., γ = 0). Removing the length normalization has the most\nnegative impact on the results. Our examination reveals that this leads to the generation of long and\nrepetitive patterns, substantially degrading the overall quality of the output (See Appendix E). Setting\nγ to 0 yields also leads to a performance degradation compared to SimPO, indicating that it is not the\noptimal target reward margin. In the following subsections, we conduct in-depth analyses to better\nunderstand both design choices.\n4.2\nLength Normalization (LN) Prevents Length Exploitation\nLN leads to an increase in the reward difference for all preference pairs, regardless of their\nlength.\nThe Bradley-Terry objective in Eq. (5) essentially aims to optimize the reward difference\n9Although our models excel on benchmarks, these evaluations have limitations, including restricted query\nspace and potential biases from model-based evaluations. Efforts like WildBench [93] aim to expand these\nspaces, where SimPO models demonstrate competitive performance.\n7\n",
  "8": "0.0\n0.8\n1.6\n2.4\nTarget Reward Margin γ\n14\n17\n20\n23\nAlpacaEval2 LC (%)\n71\n72\n73\n74\nReward Accuracy\n(a) Performance w/ different γ.\n−5\n0\n5\n10\nReward diff. r(x, yw) −r(x, yl)\n0.0\n0.1\n0.2\n0.3\n0.4\nDensity\nγ = 0.0\nγ = 0.8\nγ = 1.6\nγ = 2.4\n(b) Reward diff. distribution.\n−6\n−4\n−2\n0\nAvg. log prob. on winning pθ(yw|x)\n0.0\n0.4\n0.8\n1.2\nDensity\nγ = 0.0\nγ = 0.8\nγ = 1.6\nγ = 2.4\n(c) Log prob. distribution.\nFigure 3: Study of the margin γ. (a) Reward accuracy and AlpacaEval2 LC win rate under different\nγ values. (b) Reward difference distribution under different γ values. (c) Log likelihood distribution\non chosen responses under different γ values.\n∆r = r(x, yw) −r(x, yl) to exceed the target margin γ. We investigate the relationship between\nthe learned reward differences and the length difference ∆l = |yw| −|yl| between the winning and\nlosing responses from the training set of UltraFeedback. We measure the difference of reward (rSimPO;\nEq. (4)) using the SFT model, the SimPO model, and a model trained with SimPO but without length\nnormalization. We present the results in Figure 2a and observe that SimPO with LN consistently\nachieves a positive reward margin for all response pairs, regardless of their length difference, and\nconsistently improves the margin over the SFT model. In contrast, SimPO without LN results in a\nnegative reward difference for preference pairs when the winning response is shorter than the losing\nresponse, indicating that the model learns poorly for these instances.\nRemoving LN results in a strong positive correlation between the reward and response length,\nleading to length exploitation.\nFigures 2b and 2c illustrate the average log likelihood (pθ in Eq. (3))\nversus response length on a held-out set for models trained with SimPO and SimPO without LN. The\nmodel trained without LN exhibits a much stronger positive Spearman correlation between likelihood\nand response length compared to SimPO, indicating a tendency to exploit length bias and generate\nlonger sequences (see Appendix E). In contrast, SimPO results in a Spearman correlation coefficient\nsimilar to the SFT model (see Figure 6a).\n4.3\nThe Impact of Target Reward Margin in SimPO\nInfluence of γ on reward accuracy and win rate.\nWe investigate how the target reward margin γ\nin SimPO affects the reward accuracy on a held-out set and win rate on AlpacaEval 2, presenting\nthe results in Figure 3a. Reward accuracy is measured as the percentage of preference pairs where\nthe winning response ends up having a higher reward for the winning response than the losing\nresponse (i.e., r(x, yw) > r(x, yl)). We observe that reward accuracy increases with γ on both\nbenchmarks, indicating that enforcing a larger target reward margin effectively improves reward\naccuracy. However, the win rate on AlpacaEval 2 first increases and then decreases with γ, suggesting\nthat generation quality is not solely determined by the reward margin.\nImpact of γ on the reward distribution.\nWe visualize the distribution of the learned reward margin\nr(x, yw)−r(x, yl) and the reward of winning responses r(x, yw) under varying γ values in Figure 2b\nand Figure 2c. Notably, increasing γ tends to flatten both distributions and reduce the average log\nlikelihood of winning sequences. This initially improves performance but can eventually lead to\nmodel degeneration. We hypothesize that there is a trade-off between accurately approximating\nthe true reward distribution and maintaining a well-calibrated likelihood when setting the γ value.\nFurther exploration of this balance is deferred to future work.\n4.4\nIn-Depth Analysis of DPO vs. SimPO\nIn this section, we compare SimPO to DPO in terms of (1) likelihood-length correlation, (2) re-\nward formulation, (3) reward accuracy, and (4) algorithm efficiency. We demonstrate that SimPO\noutperforms DPO in terms of reward accuracy and efficiency.\nDPO reward implicitly facilitates length normalization.\nAlthough the DPO reward expression\nr(x, y) = β log πθ(y|x)\nπref(y|x) (with the partition function excluded) lacks an explicit term for length\nnormalization, the logarithmic ratio between the policy model and the reference model can serve to\n8\n",
  "9": "0\n200\n400\n600\n800\n1000\nResponse length |y|\n−15\n−10\n−5\n0\nAvg. log prob. pθ(y|x)\nρ = 0.59\n(a) Length correlation (DPO).\npw > pl\npw < pl\nGeneration metric\nrw < rl\nrw > rl\nTraining metric\n8.0k\n8.3k\n23.0k\n21.2k\n10\n15\n20\nx1000\n(b) Contingency table (DPO).\nMistral-Base\nMistral-Instruct\n0.4\n0.5\n0.6\n0.7\n0.8\nReward Accuracy\nDPO\nSimPO\n(c) Reward Accuracy.\nFigure 4: Comparison between SimPO and DPO, measured on UltraFeedback. (a) Spearman\ncorrelation between average log probability and response length for DPO. (b) Contingency table of\nrankings based on DPO rewards and the average log likelihood (measured on the training set). (c)\nReward accuracy of DPO and SimPO.\n0\n100\n200\n300\n400\nTraining step\n0.0\n0.2\n0.4\n0.6\nKL divergence on yw\nDPO (β = 0.01)\nDPO (β = 0.1)\nSimPO (β = 2)\nSimPO (β = 10)\n(a) KL divergence w/ different β.\n0\n10\n20\nAlpacaEval2 LC (%)\nβ = 0.1 β = 0.01\nβ = 10\nβ = 2\nDPO\nSimPO\n(b) Performance w/ different β.\nRun time\nPeak GPU memory\n0\n25\n50\n75\n100\nPercentage (%)\n73 min\n77 GB\n60 min\n69 GB\nDPO\nSimPO\n(c) Efficiency of DPO vs. SimPO.\nFigure 5: Comparison between SimPO and DPO (continued). (a) With different β in DPO and\nSimPO, KL divergence from the policy model to the reference model on yw. (b) AlpacaEval2 LC\nwin rate of DPO and SimPO with different β. (c) Runtime and memory usage for DPO and SimPO.\nimplicitly counteract length bias. As shown in Table 6 and Figure 4a, employing DPO reduces the\nSpearman correlation coefficient between average log likelihood and response length compared to\nthe approach without any length normalization (referred to as “SimPO w/o LN”). However, it still\nexhibits a stronger positive correlation when compared to SimPO.10\nTable 6: Spearman correlation ρ be-\ntween average log likelihood of differ-\nent models and response length on a\nheld-out set.\nSimPO w/o LN\nDPO\nSimPO\nρ\n0.82\n0.59\n0.34\nDPO reward mismatches generation likelihood.\nThere\nis a divergence between DPO’s reward formulation,\nrθ(x, y) = β log πθ(y|x)\nπref(y|x), and the average log likelihood\nmetric, pθ(y | x) =\n1\n|y| log πθ(y | x), which directly\nimpacts generation.\nAs shown in Figure 4b, among\nthe instances on the UltraFeedback training set where\nrθ(x, yw) > rθ(x, yl) , almost half of the pairs have\npθ(yw | x) < pθ(yl | x). In contrast, SimPO directly\nemploys the average log likelihood (scaled by β) as the reward expression, thereby eliminating the\ndiscrepancy completely, as demonstrated in Figure 6b.\nDPO lags behind SimPO in terms of reward accuracy.\nIn Figure 4c, we compare the reward\naccuracy of SimPO and DPO, assessing how well their final learned rewards align with preference\nlabels on a held-out set. SimPO consistently achieves higher reward accuracy than DPO, suggesting\nthat our reward design facilitates better generalization and leads to higher quality generations.\nKL divergence of SimPO and DPO.\nIn Figure 5a, we present the KL divergence between the\npolicy model trained with DPO and SimPO and the reference model with different β, measured\non the winning responses from a held-out set during training. Figure 5b shows the corresponding\nAlpacaEval 2 LC win rate. Although SimPO does not apply any form of regularization against\nthe reference model, the KL divergence of SimPO is reasonably small. Increasing β reduces the\nKL divergence for both DPO and SimPO, with DPO exhibiting a more pronounced reduction at\nhigher β values. In this particular setting (Mistral-base), Figure 5b demonstrates that a smaller β can\n10Note that this correlation does not fully reflect the generation length. Despite DPO showing a stronger\ncorrelation, the length of its generated responses is comparable to or even slightly shorter than those of the\nSimPO models. Please find more details in Appendix E.\n9\n",
  "10": "improve AlpacaEval 2 performance, despite the higher KL divergence.11 We hypothesize that when\nthe reference model is weak, strictly constraining the policy model to the reference model may not be\nbeneficial. As a caveat, while we did not observe any training collapse or degeneration with proper\ntuning, in principle, SimPO could potentially lead to reward hacking without explicit regularization\nagainst the reference model. In such a scenario, the model might achieve a low loss but degenerate.\nSimPO is more memory and compute-efficient than DPO.\nAnother benefit of SimPO is its\nefficiency as it does not use a reference model. Figure 5c illustrates the overall run time and per-GPU\npeak memory usage of SimPO and DPO in the Llama-3-Base setting using 8×H100 GPUs. Compared\nto a vanilla DPO implementation,12 SimPO cuts run time by roughly 20% and reduces GPU memory\nusage by about 10%, thanks to eliminating forward passes with the reference model.\n5\nRelated Work\nReinforcement learning from human feedback.\nRLHF is a technique that aligns large language\nmodels with human preferences and values [18, 102, 62, 7]. The classical RLHF pipeline typically\ncomprises three phases: supervised fine-tuning [101, 76, 33, 21, 48, 25, 82, 15, 86], reward model\ntraining [32, 60, 16, 56, 37, 50], and policy optimization [70, 4]. Proximal Policy Optimization\n(PPO) [70] is a widely used algorithm in the third stage of RLHF. The RLHF framework is also\nwidely applied to various applications, such as mitigating toxicity [3, 49, 97], ensuring safety [24],\nenhancing helpfulness [78, 83], searching and navigating the web [61], and improving model reason-\ning abilities [36]. Recently, [13] has highlighted challenges across the whole RLHF pipeline from\npreference data collection to model training. Further research has also demonstrated that RLHF can\nlead to biased outcomes, such as verbose outputs from the model [28, 71, 85].\nOffline vs. iterative preference optimization.\nGiven that online preference optimization algo-\nrithms are complex and difficult to optimize [100, 69], researchers have been exploring more efficient\nand simpler alternative offline algorithms. Direct Preference Optimization (DPO) [66] is a notable\nexample. However, the absence of an explicit reward model in DPO limits its ability to sample\npreference pairs from the optimal policy. To address this, researchers have explored augmenting\npreference data using a trained SFT policy [96] or a refined SFT policy with rejection sampling [59],\nenabling the policy to learn from data generated by the optimal policy. Further studies have extended\nthis approach to an iterative training setup, by continuously updating the reference model with the\nmost recent policy model or generating new preference pairs at each iteration [27, 46, 67, 87, 92]. In\nthis work, we focus exclusively on offline settings, avoiding any iterative training processes.\nPreference optimization objectives.\nA variety of preference optimization objectives have been\nproposed besides DPO. Ranking objectives allow for comparisons among more than two instances [26,\n58, 72, 91]. Another line of work explores simpler preference optimization objectives that do not rely\non a reference model [42, 89], similar to SimPO. [8] proposes a method to jointly optimize instructions\nand responses, finding it effectively improves DPO. [98] focuses on post-training extrapolation\nbetween the SFT and the aligned model to further enhance model performance. In this work, we\ncompare SimPO to a series of offline algorithms, including RRHF [91], SLiC-HF [96], DPO [66], IPO\n[6], CPO [88], KTO [29], ORPO [42], and R-DPO [64], and find that SimPO can outperform them\nin both efficiency and performance. Recently, [75] proposed a generalized preference optimization\nframework unifying different offline algorithms, and SimPO can be seen as a special case.\n6\nConclusion\nIn this work, we propose SimPO, a simple and effective preference optimization algorithm that\nconsistently outperforms existing approaches across various training setups. By aligning the reward\nfunction with the generation likelihood and introducing a target reward margin, SimPO eliminates\nthe need for a reference model and achieves strong performance without exploiting the length bias.\nExtensive analysis demonstrates that the key designs in SimPO are crucial and validates the efficiency\nand effectiveness of SimPO. A detailed discussion of the limitations can be found in Appendix A.\n11We observe that in some settings (e.g., Llama-3-Instruct), a large β (e.g., β = 10) leads to better performance.\n12DPO can be as memory efficient as SimPO if it were implemented to separate the forward passes of the\nreference model from the actual preference optimization. However, this implementation is not standard practice.\n10\n",
  "11": "Acknowledgments\nThe authors would like to thank Li Dong, Tianyu Gao, Tanya Goyal, Di Jin, Yuchen Lin, Kaifeng Lyu,\nSadhika Malladi, Eric Mitchell, Lewis Tunstall, Haoxiang Wang, Wei Xiong, Zhen Xu, Libing Yang,\nZhiyu Zhao, and members of the Princeton NLP group for their valuable feedback and discussions.\nWe thank Niklas Muennighoff for his advice on training and reproducing training KTO models. We\nthank Haoran Xu for helping verify our CPO runs. Mengzhou Xia is supported by an Apple Scholars\nin AIML Fellowship. This research is also funded by the National Science Foundation (IIS-2211779)\nand a Sloan Research Fellowship.\nReferences\n[1] Alan Agresti. Categorical data analysis, volume 792. John Wiley & Sons, 2012.\n[2] AI@Meta. Llama 3 model card. 2024.\n[3] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset.\narXiv preprint arXiv:2402.10571, 2024.\n[4] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning\nand tree search. Advances in neural information processing systems, 30, 2017.\n[5] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy\nJones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds,\nDanny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B.\nBrown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A general language\nassistant as a laboratory for alignment. ArXiv, abs/2112.00861, 2021.\n[6] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello,\nMichal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from\nhuman preferences. ArXiv, abs/2310.12036, 2023.\n[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn\nDrain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless\nassistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,\n2022.\n[8] Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya\nGrover. Comparing bad apples to good oranges: Aligning large language models via joint\npreference optimization. arXiv preprint arXiv:2404.00530, 2024.\n[9] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert,\nNazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.\nOpen LLM\nleaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_\nleaderboard, 2023.\n[10] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal\nmargin classifiers. In Proceedings of the fifth annual workshop on Computational learning\ntheory, pages 144–152, 1992.\n[11] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the\nmethod of paired comparisons. Biometrika, 39:324, 1952.\n[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari-\nwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language\nmodels are few-shot learners. In NeurIPS, 2020.\n[13] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier\nRando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems\nand fundamental limitations of reinforcement learning from human feedback. arXiv preprint\narXiv:2307.15217, 2023.\n11\n",
  "12": "[14] Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ran-\nganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings.\nNeurIPS, 2024.\n[15] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang,\nVijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training a better\nAlpaca with fewer data. In ICLR, 2024.\n[16] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng\nHuang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates\nhacking in RLHF. arXiv preprint arXiv:2402.07319, 2024.\n[17] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle\nLi, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al.\nChatbot arena: An open platform for evaluating llms by human preference. arXiv preprint\narXiv:2403.04132, 2024.\n[18] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep\nreinforcement learning from human preferences. Advances in neural information processing\nsystems, 30, 2017.\n[19] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick,\nand Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning\nchallenge. ArXiv, abs/1803.05457, 2018.\n[20] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,\nMatthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John\nSchulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,\n2021.\n[21] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi,\nPatrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first\ntruly open instruction-tuned LLM, 2023.\n[22] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273–297,\n1995.\n[23] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan\nLiu, and Maosong Sun. UltraFeedback: Boosting language models with high-quality feedback.\nIn ICML, 2024.\n[24] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and\nYaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint\narXiv:2310.12773, 2023.\n[25] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu,\nMaosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality\ninstructional conversations. In EMNLP, 2023.\n[26] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe\nDiao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for\ngenerative foundation model alignment. Transactions on Machine Learning Research, 2023.\n[27] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang,\nDoyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to\nonline rlhf. arXiv preprint arXiv:2405.07863, 2024.\n[28] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled\nAlpacaEval: A simple way to debias automatic evaluators. ArXiv, abs/2404.04475, 2024.\n[29] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO:\nModel alignment as prospect theoretic optimization. ArXiv, abs/2402.01306, 2024.\n12\n",
  "13": "[30] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In\nProceedings of the 56th Annual Meeting of the Association for Computational Linguistics\n(Volume 1: Long Papers), pages 889–898, 2018.\n[31] David Firth and Heather Turner. Bradley-terry models in R: the BradleyTerry2 package.\nJournal of Statistical Software, 48(9), 2012.\n[32] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.\nIn International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.\n[33] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and\nDawn Song. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023.\n[34] Ulrich Germann. Greedy decoding for statistical machine translation in almost linear time. In\nNAACL, 2003.\n[35] Alex Graves. Sequence transduction with recurrent neural networks. ArXiv, abs/1211.3711,\n2012.\n[36] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-\nYu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu.\nTeaching large language models to reason with reinforcement learning.\narXiv preprint\narXiv:2403.04642, 2024.\n[37] Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhu-\nravinskyi, Eric Hambro, and Roberta Railneau. GLoRe: When, where, and how to improve\nLLM reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024.\n[38] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and\nJacob Steinhardt. Measuring massive multitask language understanding. In International\nConference on Learning Representations, 2020.\n[39] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural\ntext degeneration. In International Conference on Learning Representations, 2019.\n[40] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi.\nLearning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting\nof the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638–1649,\n2018.\n[41] Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form\ncompetition: Why the highest probability answer isn’t always right. In Proceedings of the\n2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051,\n2021.\n[42] Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization\nwithout reference model. ArXiv, abs/2403.07691, 2024.\n[43] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou\nWang, and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via a\nhuman-preference dataset. ArXiv, abs/2307.04657, 2023.\n[44] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven-\ndra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample,\nLucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao,\nThibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. ArXiv,\nabs/2310.06825, 2023.\n[45] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language\nmodels with pairwise ranking and generative fusion. In ACL, 2023.\n[46] Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and\nChanjun Park. sDPO: Don’t use your data all at once. ArXiv, abs/2403.19270, 2024.\n13\n",
  "14": "[47] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv\npreprint arXiv:1412.6980, 2014.\n[48] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith\nStevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Ope-\nnassistant conversations-democratizing large language model alignment. In Thirty-seventh\nConference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.\n[49] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley,\nJason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human\npreferences. In International Conference on Machine Learning, pages 17506–17533. PMLR,\n2023.\n[50] Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda,\nBill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi,\nNoah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language\nmodeling. ArXiv, abs/2403.13787, 2024.\n[51] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable\nagent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871,\n2018.\n[52] Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge.\nIn Thirteenth international conference on the principles of knowledge representation and\nreasoning, 2012.\n[53] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep\nreinforcement learning for dialogue generation. In Jian Su, Kevin Duh, and Xavier Carreras,\neditors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language\nProcessing, pages 1192–1202, Austin, Texas, November 2016. Association for Computational\nLinguistics.\n[54] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and\nIon Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline, April 2024.\n[55] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin,\nPercy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-\nfollowing models. https://github.com/tatsu-lab/alpaca_eval, 2023.\n[56] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee,\nJan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv\npreprint arXiv:2305.20050, 2023.\n[57] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic\nhuman falsehoods. In ACL, pages 3214–3252, 2022.\n[58] Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mo-\nhammad Saleh, Simon Baumgartner, Jialu Liu, et al. LiPO: Listwise preference optimization\nthrough learning-to-rank. arXiv preprint arXiv:2402.01878, 2024.\n[59] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and\nJialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth\nInternational Conference on Learning Representations, 2024.\n[60] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo\nGeng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering math-\nematical reasoning for large language models via reinforced evol-instruct. arXiv preprint\narXiv:2308.09583, 2023.\n[61] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo-\npher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted\nquestion-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.\n14\n",
  "15": "[62] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin,\nChong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton,\nFraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis\nChristiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions\nwith human feedback. In NeurIPS, 2022.\n[63] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin\nWhite. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv\npreprint arXiv:2402.13228, 2024.\n[64] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from\nquality in direct preference optimization. ArXiv, abs/2403.19159, 2024.\n[65] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.\nLanguage models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.\n[66] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and\nChelsea Finn. Direct preference optimization: Your language model is secretly a reward model.\nIn NeurIPS, 2023.\n[67] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and\nTengyang Xie. Direct nash optimization: Teaching language models to self-improve with\ngeneral preferences. ArXiv, abs/2404.03715, 2024.\n[68] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint\narXiv:1609.04747, 2016.\n[69] Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient RLHF:\nReducing the memory usage of PPO. arXiv preprint arXiv:2309.00754, 2023.\n[70] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal\npolicy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n[71] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating\nlength correlations in RLHF. arXiv preprint arXiv:2310.03716, 2023.\n[72] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.\nPreference ranking optimization for human alignment. In AAAI, 2024.\n[73] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec\nRadford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback.\nAdvances in Neural Information Processing Systems, 33:3008–3021, 2020.\n[74] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene\nTarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understand-\ning the performance gap between online and offline alignment algorithms. arXiv preprint\narXiv:2405.08448, 2024.\n[75] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark\nRowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot.\nGeneralized preference optimization: A unified approach to offline alignment. arXiv preprint\narXiv:2402.05749, 2024.\n[76] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin,\nPercy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama\nmodel, 2023.\n[77] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin,\nSurya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé,\net al.\nGemma 2: Improving open language models at a practical size.\narXiv preprint\narXiv:2408.00118, 2024.\n15\n",
  "16": "[78] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-\ntuning language models for factuality. In The Twelfth International Conference on Learning\nRepresentations, 2024.\n[79] Hoang Tran, Chris Glaze, and Braden Hancock. Iterative DPO alignment. Technical report,\nSnorkel AI, 2023.\n[80] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes\nBelkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan\nSarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation\nof LM alignment. ArXiv, abs/2310.16944, 2023.\n[81] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian\nXu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika,\nDan Hendrycks, Zi-Han Lin, Yuk-Kit Cheng, Sanmi Koyejo, Dawn Xiaodong Song, and Bo Li.\nDecodingTrust: A comprehensive assessment of trustworthiness in GPT models. In NeurIPS,\n2023.\n[82] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. OpenChat:\nAdvancing open-source language models with mixed-quality data. In ICLR, 2024.\n[83] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and\nTong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference\nalignment with multi-objective rewards. ArXiv, abs/2402.18571, 2024.\n[84] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable\npreferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint\narXiv:2406.12845, 2024.\n[85] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu,\nDavid Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go?\nexploring the state of instruction tuning on open resources. In Thirty-seventh Conference on\nNeural Information Processing Systems Datasets and Benchmarks Track, 2023.\n[86] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS:\nSelecting influential data for targeted instruction tuning. In ICML, 2024.\n[87] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong\nZhang. Iterative preference learning from human feedback: Bridging theory and practice for\nrlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.\n[88] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme,\nKenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the\nboundaries of LLM performance in machine translation. ArXiv, abs/2401.08417, 2024.\n[89] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more\ncringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint\narXiv:2312.16682, 2023.\n[90] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao\nYu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. arXiv\npreprint arXiv:2404.10719, 2024.\n[91] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF:\nRank responses to align language models with human feedback. In NeurIPS, 2023.\n[92] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and\nJason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.\n[93] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander,\nValentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking\nLLMs with challenging tasks from real users in the wild. arXiv e-prints, pages arXiv–2406,\n2024.\n16\n",
  "17": "[94] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a\nmachine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, edi-\ntors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,\npages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.\n[95] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat:\n1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning\nRepresentations, 2024.\n[96] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu.\nSLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425,\n2023.\n[97] Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation\nwith sequence likelihood contrastive learning. In Findings of ACL, 2023.\n[98] Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapo-\nlation expedites alignment. arXiv preprint arXiv:2404.16792, 2024.\n[99] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao\nZhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with\nMT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023.\n[100] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie\nJin, Qin Liu, Yuhao Zhou, et al. Secrets of RLHF in large language models part I: PPO. arXiv\npreprint arXiv:2307.04964, 2023.\n[101] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma,\nAvia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. NeurIPS, 2023.\n[102] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei,\nPaul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.\narXiv preprint arXiv:1909.08593, 2019.\n17\n",
  "18": "A\nLimitations\nMore in-depth theoretical analysis. Despite the empirical success and intuitive motivation of\nSimPO, a more rigorous theoretical analysis is necessary to fully understand the factors contributing\nto its effectiveness. Additionally, we introduce an additional hyperparameter, the target reward\nmargin, which requires manual tuning. Future work could explore how to determine the optimal\nmargin automatically and provide a more theoretical understanding of SimPO.\nSafety and honesty. SimPO is designed to optimize the generation quality of language models\nby pushing the margin between the average log likelihood of the winning response and the losing\nresponse to exceed a target reward margin. However, it does not explicitly consider safety and honesty\naspects, which are crucial for real-world applications. Future work should explore integrating safety\nand honesty constraints into SimPO to ensure that the generated responses are not only high-quality\nbut also safe and honest. The dataset used in this work, UltraFeedback [23], primarily focuses on\nhelpfulness, and future research may consider a more comprehensive study utilizing larger-scale\npreference datasets [43, 95] and evaluation benchmarks [81] that place a strong emphasis on safety\naspects. Nonetheless, we observe that this method consistently achieves high TruthfulQA [57]\nperformance compared to other objectives in Table 9, suggesting its potential for safety alignment.\nPerformance drop on math. We observed that preference optimization algorithms generally\ndecrease downstream task performance, particularly on reasoning-heavy tasks like GSM8k, as shown\nin Table 9. SimPO occasionally results in performance comparable to or worse than DPO. We\nhypothesize that this may be related to the choice of training datasets, hyperparameters used for\ntraining, or a mismatch of chat templates used for downstream task evaluations. One explanation is\nthat the preference optimization objective may not be effectively increasing the likelihood of preferred\nsequences despite increasing the reward margin. [63] first observed this phenomenon and point out\nthat this can hinder learning from math preference pairs where changing one token can flip the label\n(e.g., changing 2 + 2 = 4 to 2 + 2 = 5). They propose a simple regularization strategy to add back a\nreference-model calibrated supervised fine-tuning loss to the preference optimization objective, and\neffectively mitigate this issue. Future work may consider integrating this regularization strategy into\nSimPO to improve performance on reasoning-heavy tasks.\nB\nImplementation Details\nWe find that hyperparameter tuning is crucial for achieving optimal performance of preference\noptimization methods. However, the importance of careful hyperparameter tuning may have been\nunderestimated in prior research, potentially leading to suboptimal baseline results. To ensure a\nfair comparison, we conduct thorough hyperparameter tuning for all methods compared in our\nexperiments.\nTable 8: The hyperparameter values in SimPO\nused for each training setting.\nSetting\nβ\nγ\nLearning rate\nMistral-Base\n2.0\n1.6\n3e-7\nMistral-Instruct\n2.5\n0.3\n5e-7\nLlama-3-Base\n2.0\n1.0\n6e-7\nLlama-3-Instruct\n2.5\n1.4\n1e-6\nGeneral training hyperparameters.\nFor the\nBase training setups, we train SFT models using\nthe UltraChat-200k dataset [25] with the following\nhyperparameters: a learning rate of 2e-5, a batch\nsize of 128, a max sequence length of 2048, and\na cosine learning rate schedule with 10% warmup\nsteps for 1 epoch. All the models are trained with\nan Adam optimizer [47].\nFor the preference optimization stage, we conduct\npreliminary experiments to search for batch sizes in [32, 64, 128] and training epochs in [1, 2, 3].\nWe find that a batch size of 128 and a single training epoch generally yield the best results across all\nmethods. Therefore, we fix these values for all preference optimization experiments. Additionally, we\nset the max sequence length to be 2048 and apply a cosine learning rate schedule with 10% warmup\nsteps on the preference optimization dataset.\nMethod-specific training hyperparameters.\nWe have noticed that the optimal learning rate varies\nfor different preference optimization methods and greatly influences the benchmark performance.\nTherefore, we individually search the learning rates in the range of [3e-7, 5e-7, 6e-7, 1e-6] for each\n18\n",
  "19": "Table 7: Various preference optimization objectives and hyperparameter search range.\nMethod\nObjective\nHyperparameter\nRRHF [91]\nmax\n\u0010\n0, −\n1\n|yw| log πθ(yw|x) +\n1\n|yl| log πθ(yl|x)\n\u0011\n−λ log πθ(yw|x)\nλ ∈[0.1, 0.5, 1.0, 10.0]\nSLiC-HF [96]\nmax (0, δ −log πθ(yw|x) + log πθ(yl|x)) −λ log πθ(yw|x)\nλ ∈[0.1, 0.5, 1.0, 10.0]\nβ ∈[0.1, 0.5, 1.0, 2.0]\nDPO [66]\n−log σ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −β log πθ(yl|x)\nπref(yl|x)\n\u0011\nβ ∈[0.01, 0.05, 0.1]\nIPO [6]\n\u0010\nlog πθ(yw|x)\nπref(yw|x) −log πθ(yl|x)\nπref(yl|x) −\n1\n2τ\n\u00112\nτ ∈[0.01, 0.1, 0.5, 1.0]\nCPO [88]\n−log σ (β log πθ(yw|x) −β log πθ(yl|x)) −λ log πθ(yw|x)\nλ = 1.0, β ∈[0.01, 0.05, 0.1]\nKTO [29]\n−λwσ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −zref\n\u0011\n+ λlσ\n\u0010\nzref −β log πθ(yl|x)\nπref(yl|x)\n\u0011\n,\nλl = λw = 1.0\nwhere zref = E(x,y)∼D [βKL (πθ(y|x)||πref(y|x))]\nβ ∈[0.01, 0.05, 0.1]\nORPO [42]\n−log pθ(yw|x) −λ log σ\n\u0010\nlog\npθ(yw|x)\n1−pθ(yw|x) −log\npθ(yl|x)\n1−pθ(yl|x)\n\u0011\n,\nλ ∈[0.1, 0.5, 1.0, 2.0]\nwhere pθ(y|x) = exp\n\u0010\n1\n|y| log πθ(y|x)\n\u0011\nR-DPO [64]\n−log σ\n\u0010\nβ log πθ(yw|x)\nπref(yw|x) −β log πθ(yl|x)\nπref(yl|x) + (α|yw| −α|yl|)\n\u0011\nα ∈[0.05, 0.1, 0.5, 1.0]\nβ ∈[0.01, 0.05, 0.1]\nSimPO\n−log σ\n\u0010\nβ\n|yw| log πθ(yw|x) −\nβ\n|yl| log πθ(yl|x) −γ\n\u0011\nβ ∈[2.0, 2.5]\nγ ∈[0.3, 0.5, 1.0, 1.2, 1.4, 1.6]\n0\n200\n400\n600\n800\n1000\nResponse length |y|\n−15\n−10\n−5\n0\nAvg. log prob. pθ(y|x)\nρ = 0.33\n(a) Mistral-SFT.\npw > pl\npw < pl\nGeneration metric\nrw < rl\nrw > rl\nTraining metric\n0\n13.8k\n46.7k\n0\n0\n20\n40\nx1000\n(b) Contingency table (SimPO).\nFigure 6: (a) Likelihood-length correlation plot for Mistral-SFT fine-tuned on UltraChat-200k. (a)\nContingency table rankings based on SimPO rewards and the average log likelihood (measured on\nthe training set).\nmethod. Table 7 shows the detailed information on method-specific hyperparameters search ranges\nfor baselines.13 Table 8 shows SimPO’s hyperparameters used under each setting.\nDecoding hyperparameters.\nFor AlpacaEval 2, we use a sampling decoding strategy to generate\nresponses, with a temperature of 0.7 for the Mistral-Base setting following zephyr-7b-beta,14 a\ntemperature of 0.5 for the Mistral-Instruct setting following Snorkel-Mistral-PairRM-DPO,\nand a temperature of 0.9 for both Llama 3 settings.15 For Arena-Hard, we use the default greedy\ndecoding for all settings and methods. For MT-Bench, we follow the official decoding configuration\nwhich defines different sampling temperatures for different categories.\nComputation environment.\nAll the training experiments in this paper were conducted on 8×H100\nGPUs based on the alignment-handbook repo.16\n13There is a discrepancy between the KTO runs in their original paper, where the original runs use a RMSProp\noptimizer [68]. We use an Adam optimizer [47] for all the experiments.\n14https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/\nmodels_configs/zephyr-7b-beta/configs.yaml\n15We grid search the temperature hyperparameter for the Llama-3-Base setting with DPO over\n0.1, 0.3, 0.5, 0.7, 0.9, and fix it for all different methods.\n16https://github.com/huggingface/alignment-handbook\n19\n",
  "20": "C\nDownstream Task Evaluation\nTo examine how preference optimization methods affect downstream task performance, we evaluate\nmodels trained with different methods on various tasks listed on the Huggingface Open Leader-\nboard [9]. These tasks include MMLU [38], ARC [19], HellaSwag [94], TruthfulQA [57], Wino-\ngrad [52], and GSM8K [20]. We follow the established evaluation protocols and present the results\nfor all models in Table 9. Generally, we find that preference optimization’s effect varies across tasks.\nKnowledge is largely retained with a small loss.\nCompared to the SFT checkpoint, we find that\nall preference optimization methods generally maintain MMLU performance with minimal decline.\nIn this aspect, SimPO is largely comparable to DPO.\nReading comprehension and commonsense reasoning improves.\nFor ARC and HellaSwag,\npreference optimization methods generally improve performance compared to the SFT checkpoint.\nOne hypothesis is that the preference optimization dataset contains similar prompts to these tasks,\nwhich helps the model better understand the context and improve reading comprehension and\ncommonsense reasoning abilities.\nTruthfulness improves.\nSurprisingly, we find that preference optimization methods consistently\nimprove TruthfulQA performance compared to the SFT checkpoint, and the improvement could be\nas high as over 10% in some cases. Similarly, we hypothesize that the preference dataset contains\ninstances that emphasize truthfulness, which helps the model better understand the context and\ngenerate more truthful responses.\nMath performance drops.\nGSM8K is the benchmark that shows the most volatility across methods.\nNotably, except for ORPO, almost all approaches lead to consistent drops in one or more settings.\nWe hypothesize that ORPO retains performance largely due to its supervised fine-tuning loss for\nregulation. [63] adds a reference-model calibrated supervised fine-tuning loss to the preference\noptimization objective, and find that it effectively solves the issue and maintains performance on\nmath tasks as well.\nOverall, identifying a pattern in downstream performance is challenging. Comprehensive analysis is\ndifficult due to using different pretrained models, preference optimization datasets, and objectives.\nRecent works indicate that gradient-based approaches could be effective in finding relevant data\nfor downstream tasks [86], and could possibly extended to understand the effect of preference\noptimization. We believe a thorough study on how preference optimization affects downstream\nperformance would be valuable and call for a rigorous and more comprehensive analysis in future\nwork.\nD\nStandard Deviation of AlpacaEval 2 and Arena-Hard\nWe present the standard deviation of AlpacaEval 2 and the 95% confidence interval of Arena-Hard in\nTable 10. All these metrics are reasonable and do not exhibit any significant outliers or instability.\nE\nGeneration Length Analysis\nLength normalization decreases generation length and improves generation quality.\nRemov-\ning length normalization from the SimPO objective results in an approach similar to Contrastive\nPreference Optimization (CPO) [88], which interpolates reward maximization with a supervised\nfine-tuning loss and has demonstrated strong performance in machine translation. However, without\nthe supervised fine-tuning loss, the reward maximization objective without length normalization is\nsuboptimal in preference optimization.\nWe analyze the generation length of models trained with or without length normalization on Al-\npacaEval 2 and Arena-Hard. As shown in Figure 6, length normalization significantly decrease the\ngeneration length by up to 25% compared to when it is not used in most cases. However, even though\nthe generation length is shorter, the models with length normalization consistently achieve much\nhigher win rates on both benchmarks. This suggests that length normalization can effectively control\nthe verbosity of the generated responses, and meanwhile improve the generation quality.\n20\n",
  "21": "Table 9: Downstream task evaluation results of tasks on the huggingface open leaderboard.\nMMLU (5)\nARC (25)\nHellaSwag (10)\nTruthfulQA (0)\nWinograd (5)\nGSM8K (5)\nAverage\nMistral-Base\nSFT\n60.10\n58.28\n80.76\n40.35\n76.40\n28.13\n57.34\nRRHF\n57.41\n52.13\n80.16\n43.73\n76.64\n4.78\n52.48\nSLiC-HF\n59.24\n55.38\n81.15\n48.36\n77.35\n33.74\n59.20\nDPO\n58.48\n61.26\n83.59\n53.06\n76.80\n21.76\n59.16\nIPO\n60.23\n60.84\n83.30\n45.44\n77.58\n27.14\n59.09\nCPO\n59.39\n57.00\n80.75\n47.07\n76.48\n33.06\n58.96\nKTO\n60.90\n62.37\n84.88\n56.60\n77.27\n38.51\n63.42\nORPO\n63.20\n61.01\n84.09\n47.91\n78.61\n42.15\n62.83\nR-DPO\n59.58\n61.35\n84.29\n46.12\n76.56\n18.12\n57.67\nSimPO\n59.21\n62.63\n83.60\n50.68\n77.27\n22.21\n59.27\nMistral-Instruct\nSFT\n60.40\n63.57\n84.79\n66.81\n76.64\n40.49\n65.45\nRRHF\n59.75\n64.42\n85.54\n67.98\n76.64\n37.76\n65.35\nSLiC-HF\n60.59\n59.90\n84.05\n65.30\n76.32\n39.65\n64.30\nDPO\n60.53\n65.36\n85.86\n66.71\n76.80\n40.33\n65.93\nIPO\n60.20\n63.31\n84.88\n67.36\n75.85\n39.42\n65.17\nCPO\n60.36\n63.23\n84.47\n67.38\n76.80\n38.74\n65.16\nKTO\n60.52\n65.78\n85.49\n68.45\n75.93\n38.82\n65.83\nORPO\n60.43\n61.43\n84.32\n66.33\n76.80\n36.85\n64.36\nR-DPO\n60.71\n66.30\n86.01\n68.22\n76.72\n37.00\n65.82\nSimPO\n60.53\n66.89\n85.95\n68.40\n76.32\n35.25\n65.56\nLlama-3-Base\nSFT\n64.88\n60.15\n81.37\n45.33\n75.77\n46.32\n62.30\nRRHF\n64.71\n62.12\n82.03\n55.01\n77.51\n44.28\n64.27\nSLiC-HF\n64.36\n61.43\n81.88\n54.95\n77.27\n48.82\n64.79\nDPO\n64.31\n64.42\n83.87\n53.48\n76.32\n38.67\n63.51\nIPO\n64.40\n62.88\n80.46\n54.20\n72.22\n22.67\n59.47\nCPO\n64.98\n61.69\n82.03\n54.29\n76.16\n46.93\n64.35\nKTO\n64.42\n63.14\n83.55\n55.76\n76.09\n38.97\n63.65\nORPO\n64.44\n61.69\n82.24\n56.11\n77.51\n50.04\n65.34\nR-DPO\n64.19\n64.59\n83.90\n53.41\n75.93\n39.27\n63.55\nSimPO\n64.00\n65.19\n83.09\n59.46\n77.19\n31.54\n63.41\nLlama-3-Instruct\nSFT\n67.06\n61.01\n78.57\n51.66\n74.35\n68.69\n66.89\nRRHF\n67.20\n61.52\n79.54\n53.76\n74.19\n66.11\n67.05\nSLiC-HF\n66.41\n61.26\n78.80\n53.23\n76.16\n66.57\n67.07\nDPO\n66.88\n63.99\n80.78\n59.01\n74.66\n49.81\n65.86\nIPO\n66.52\n61.95\n77.90\n54.64\n73.09\n58.23\n65.39\nCPO\n67.05\n62.29\n78.73\n54.01\n73.72\n67.40\n67.20\nKTO\n66.38\n63.57\n79.51\n58.15\n73.40\n57.01\n66.34\nORPO\n66.41\n61.01\n79.38\n54.37\n75.77\n64.59\n66.92\nR-DPO\n66.74\n64.33\n80.97\n60.32\n74.82\n43.90\n65.18\nSimPO\n65.63\n62.80\n78.33\n60.70\n73.32\n50.72\n65.25\nLength is not a reliable indicator of generation quality.\nWe further analyze the generation length\nof models trained with different methods on AlpacaEval 2 and Arena-Hard, as shown in Table 10.\nGenerally, we find that no single method consistently generates longer or shorter responses across\nall settings. Additionally, even though some methods may generate longer responses, they do not\nnecessarily achieve better win rates on the benchmarks. This indicates that the length of the generated\nresponses is not a reliable indicator of generation quality.\nSimPO demonstrates minimal exploitation of response length.\nWe observe that SimPO has\na shorter generation length compared to DPO in the Llama-3-Instruct case but exhibits a higher\ngeneration length in other settings, with up to 26% longer responses on AlpacaEval 2. Conversely,\nSimPO only increases length by only around 5% on Arena-Hard compared to DPO. It is fair to say\nthat the generation length heavily depends on the evaluation benchmark. A stronger indicator is that\nSimPO consistently achieves a higher length-controlled win rate on AlpacaEval 2 compared to the\nraw win rate, demonstrating minimal exploitation of response length.\n21\n",
  "22": "A < B\nA >= B\nA = SimPO\nA >= B\nA < B\nA = DPO\n2.6%\n9.2%\n78.6%\n9.6%\n0\n20\n40\n60\n80\n100\n%\n(a) Mistral-Base.\nA < B\nA >= B\nA = SimPO\nA >= B\nA < B\nA = DPO\n5.7%\n17.9%\n56.9%\n19.5%\n0\n20\n40\n60\n80\n100\n%\n(b) Mistral-Instruct.\nFigure 7: Win rate heatmap of Mistral-Base and Mistral-Instruct on AlpacaEval 2. B represents the\nbaseline model (i.e., GPT-4-Preview-1106).\nA << B\nA < B\nA = B\nA > B\nA >> B\nA = SimPO\nA >> B\nA > B\nA = B\nA < B\nA << B\nA = DPO\n0.5%\n0.5%\n0.0%\n0.3%\n0.0%\n3.6%\n3.8%\n0.8%\n2.9%\n0.4%\n1.3%\n2.5%\n0.9%\n1.4%\n0.6%\n12.9%\n16.3%\n2.0%\n9.2%\n0.9%\n18.5%\n13.0%\n1.2%\n5.9%\n0.6%\n0\n5\n10\n15\n20\n%\n(a) Mistral-Base.\nA << B\nA < B\nA = B\nA > B\nA >> B\nA = SimPO\nA >> B\nA > B\nA = B\nA < B\nA << B\nA = DPO\n0.4%\n0.4%\n0.1%\n0.7%\n0.3%\n4.2%\n6.1%\n1.3%\n6.1%\n0.6%\n2.1%\n3.0%\n0.6%\n2.5%\n0.2%\n10.3%\n13.6%\n2.5%\n8.8%\n1.5%\n13.4%\n11.1%\n2.9%\n6.7%\n0.6%\n0\n5\n10\n15\n20\n%\n(b) Mistral-Instruct.\nFigure 8: Win rate heatmap of Mistral-Base and Mistral-Instruct on Arena-Hard. B represents the\nbaseline model (i.e., GPT-4-0314).\nF\nGradient Analysis\nWe examine the gradients of SimPO and DPO to understand their different impact on the training\nprocess.\n∇θLSimPO(πθ) = −βE(x,yw,yl)∼D\n\nsθ ·\n\n\n\n\n\n1\n|yw|∇θ log πθ(yw|x)\n|\n{z\n}\nincrease likelihood on yw\n−1\n|yl|∇θ log πθ(yl|x)\n|\n{z\n}\ndecrease likelihood on yl\n\n\n\n\n\n\n,\n∇θLDPO(πθ) = −βE(x,yw,yl)∼D\n\ndθ ·\n\n\n∇θ log πθ(yw|x)\n|\n{z\n}\nincrease likelihood on yw\n−∇θ log πθ(yl|x)\n|\n{z\n}\ndecrease likelihood on yl\n\n\n\n\n,\n(7)\nwhere\nsθ = σ\n\u0012 β\n|yl| log πθ(yl|x) −\nβ\n|yw| log πθ(yw|x) + γ\n\u0013\n,\ndθ = σ\n\u0012\nβ log πθ(yl|x)\nπref(yl|x) −β log πθ(yw|x)\nπref(yw|x)\n\u0013\nrepresent the gradient weight in SimPO and DPO, respectively. It can be seen that the differences\nare twofold: (1) comparing the gradient weights sθ and dθ, SimPO’s gradient weight sθ does not\ninvolve the reference model and has a straightforward interpretation: the weights will be higher for\nsamples where the policy model incorrectly assigns higher likelihood to yl than yw; (2) comparing\nthe gradient updates, SimPO’s gradients on yl and yw are length-normalized, while DPO’s are not.\nThis corresponds to the empirical findings [64] that DPO may exploit length bias: longer sequences\nwith more tokens will receive larger gradient updates in DPO, dominating the training process.\n22\n",
  "23": "G\nQualitative Analysis\nWe present the win rate heatmap of Mistral-Base and Mistral-Instruct on AlpacaEval 2 and Arena-\nHard in Figure 7 and Figure 8, respectively. Based on this analysis, we present qualitative examples of\nresponses generated by a SimPO model, a DPO model and the baseline model GPT-4-Preview-1106\non AlpacaEval 2.\nComparing SimPO with DPO.\nIn Figure 9 and Figure 10, we present an example where Mistral-\nBase-SimPO generates a better-structured answer compared to Mistral-Base-DPO. Given the question,\n\"How can you determine if a person is genuinely interested in a conversation or simply being polite?\",\nthe DPO model generates a response with a long list of bullet points, making it difficult to understand\nthe relationships between different points. In contrast, the SimPO model produces a well-structured\nanswer with high-level categorization of different behaviors, followed by detailed suggestions for\neach category. This makes the answer more readable and easier to understand.\nComparing Instruct models with Base models when trained with SimPO.\nIn Figure 11, we\npresent an example where Llama-3-Instruct generates a more detailed and well-formatted answer\ncompared to the baseline model, and as well as the Llama-3-Base-SimPO model. Given the question:\nWhat language does Argentina people speak? Llama-3-Base-SimPO only gives a very brief answer.\nGPT-4-Preview-1106 gives a more detailed answer in explaining how the Argentina Spanish differs\nfrom standard Spanish. However, the answer is not well formatted and a bit hard to parse. Llama-3-\nInstruct-SimPO gives a detailed and well-formatted answer, which is easier to read and understand,\nand offers sufficient details.\n23\n",
  "24": "Figure 9: An AlpacaEval 2 generation from the MistralBase model after training with DPO.\nInput →\nQ: How can you determine if a person is genuinely\ninterested in a conversation or simply being\npolite?\nMistral-Base DPO →\nAs an AI, I don´t have access to contextual\ninformation or nonverbal cues, but here are some\ngeneral signs that may indicate whether someone\nis genuinely interested in a conversation or being\npolite:\n1.\nActive listening:\nA genuinely interested\nperson will actively listen to what you´re saying,\nask questions, and provide feedback or comments.\nThey may also maintain eye contact, nod, or use\nverbal cues like \"uh-huh\" or \"I see\" to show\nthey´re engaged.\n2.\nFollow-up questions:\nA genuine interest\nin the conversation often leads to follow-up\nquestions.\nIf someone is asking questions that\nshow they´re interested in learning more about the\ntopic, it´s a good sign they´re engaged.\n3.\nBody language:\nNonverbal cues like leaning\nin, smiling, and mirroring your body language can\nindicate genuine interest.\nHowever, note that\nsome people may have different body language\nstyles, so this may not always be a reliable\nindicator.\n4.\nTopic continuation:\nIf the conversation flows\nnaturally, and both parties are contributing\nequally, it´s a sign of mutual interest.\nIf\none person dominates the conversation or keeps\nsteering it back to their own interests, it may\nindicate a lack of genuine interest.\n5.\nTime spent:\nIf the conversation lasts longer\nthan expected or planned, it´s a sign that both\nparties are engaged and interested.\n6.\nFollow-up actions:\nIf someone shows genuine\ninterest, they may follow up with you after the\nconversation, whether it´s through a message,\nemail, or phone call.\nRemember, however, that sometimes people may\nbe polite or professional, even if they´re not\ngenuinely interested.\nIt´s essential to consider\nthe context and the relationship you have with\nthe person to determine their level of interest\naccurately.\n24\n",
  "25": "Table 10: Detailed results of AlpacaEval 2 and Arena-Hard. LC means length-controlled win rate,\nWR means raw win rate, and STD means standard deviation of win rate. Length is the average\ngeneration length. For Arena-Hard, we report the win rate and 95% confidence interval.\nAlpacaEval 2\nArena-Hard\nModels\nLC (%)\nWR (%)\nSTD (%)\nLength\nWR\n95 CI high\n95 CI low\nLength\nMistral-Base\nSFT\n8.4\n6.2\n1.1\n914\n1.3\n1.8\n0.9\n521\nRRHF\n11.6\n10.2\n0.9\n1630\n6.9\n8.0\n6.0\n596\nSLiC-HF\n10.9\n8.9\n0.9\n1525\n7.3\n8.5\n6.2\n683\nDPO\n15.1\n12.5\n1.0\n1477\n10.4\n11.7\n9.4\n628\nIPO\n11.8\n9.4\n0.9\n1380\n7.5\n8.5\n6.5\n674\nCPO\n9.8\n8.9\n0.9\n1827\n5.8\n6.7\n4.9\n823\nKTO\n13.1\n9.1\n0.9\n1144\n5.6\n6.6\n4.7\n475\nORPO\n14.7\n12.2\n1.0\n1475\n7.0\n7.9\n5.9\n764\nR-DPO\n17.4\n12.8\n1.0\n1335\n9.9\n11.1\n8.4\n528\nSimPO\n21.4\n20.8\n1.2\n1868\n16.6\n18.0\n15.1\n699\nMistral-Instruct\nSFT\n17.1\n14.7\n1.1\n1676\n12.6\n14.1\n11.1\n486\nRRHF\n25.3\n24.8\n1.3\n1927\n18.1\n19.5\n16.4\n517\nSLiC-HF\n24.1\n24.6\n1.3\n2088\n18.9\n20.6\n17.3\n578\nDPO\n26.8\n24.9\n1.3\n1808\n16.3\n18.0\n15.2\n518\nIPO\n20.3\n20.3\n1.2\n2024\n16.2\n17.9\n14.4\n740\nCPO\n23.8\n28.8\n1.3\n3245\n22.6\n25.0\n20.8\n812\nKTO\n24.5\n23.6\n1.3\n1901\n17.9\n20.3\n16.1\n496\nORPO\n24.5\n24.9\n1.3\n2022\n20.8\n22.5\n19.1\n527\nR-DPO\n27.3\n24.5\n1.3\n1784\n16.1\n18.0\n14.6\n495\nSimPO\n32.1\n34.8\n1.4\n2193\n21.0\n22.7\n18.8\n539\nLlama-3-Base\nSFT\n6.2\n4.6\n0.7\n1082\n3.3\n4.0\n2.6\n437\nRRHF\n10.8\n8.1\n0.9\n1186\n6.6\n7.5\n5.7\n536\nSLiC-HF\n12.1\n10.1\n0.9\n1540\n10.3\n11.5\n8.9\n676\nDPO\n18.2\n15.5\n1.1\n1585\n15.9\n18.1\n14.1\n563\nIPO\n14.4\n14.2\n1.1\n1856\n17.8\n19.5\n16.0\n608\nCPO\n12.3\n13.7\n1.0\n2495\n11.6\n13.2\n10.4\n800\nKTO\n14.2\n12.4\n1.0\n1646\n12.5\n14.2\n10.9\n519\nORPO\n12.2\n10.6\n0.9\n1628\n10.8\n12.3\n9.6\n639\nR-DPO\n17.6\n14.4\n1.1\n1529\n17.2\n18.5\n15.7\n527\nSimPO\n22.0\n20.3\n1.2\n1795\n23.4\n25.4\n21.6\n704\nLlama-3-Instrct\nSFT\n26.0\n25.3\n1.3\n1920\n22.3\n23.9\n20.3\n596\nRRHF\n31.3\n28.4\n1.33\n1805\n26.5\n28.4\n24.6\n502\nSLiC-HF\n26.9\n27.5\n1.3\n1977\n26.2\n28.4\n24.4\n584\nDPO\n40.3\n37.9\n1.4\n1883\n32.6\n34.8\n30.3\n528\nIPO\n35.6\n35.6\n1.4\n1983\n30.5\n32.8\n28.4\n554\nCPO\n28.9\n32.2\n1.4\n2166\n28.8\n30.6\n26.6\n624\nKTO\n33.1\n31.8\n1.4\n1909\n26.4\n28.7\n24.3\n536\nORPO\n28.5\n27.4\n1.3\n1888\n25.8\n27.4\n23.8\n535\nR-DPO\n41.1\n37.8\n1.4\n1854\n33.1\n35.3\n30.9\n522\nSimPO\n44.7\n40.5\n1.4\n1825\n33.8\n35.9\n32.0\n504\n25\n",
  "26": "Table 11: Average response lengths on AlpacaEval 2 and Arena-Hard trained with Mistral-Base or\nMistral-Instruct.\nModel\nAlpacaEval 2\nArena-Hard\nMistral-Base\nMistral-Instruct\nMistral-Base\nMistral-Instruct\nLC (%)\nLength\nLC (%)\nLength\nWR (%)\nLength\nWR (%)\nLength\nSimPO\n21.5\n1868\n32.1\n2193\n16.6\n699\n21.0\n539\nSimPO w/o LN\n11.9\n2345\n19.1\n2067\n9.4\n851\n16.3\n679\n26\n",
  "27": "Figure 10: An AlpacaEval 2 generation from the Mistral-Base model after training with SimPO.\nCompared to the output generated by the DPO model, as shown in Figure 9, the generation by SimPO\nis better structured with hierarchical discussions, making the information more clearly presented and\nreadable.\nInput →\nQ: How can you determine if a person is\ngenuinely interested in a conversation or simply\nbeing polite?\nMistral-Base SimPO →\nHere are some indicators to consider:\n1.\nEngagement:\n- Active listening:\nDoes the person maintain\neye contact, nod, ask follow-up questions, and\nrespond thoughtfully to what you say?\nEngaged\nindividuals tend to show genuine interest by\ndemonstrating attentiveness and participation in\nthe conversation.\n- Initiative:\nDoes the person take the lead\nin steering the conversation, expressing\nenthusiasm, sharing personal experiences, or\nasking questions beyond polite small talk?\nSomeone who genuinely cares about the topic or\nconnection may show initiative in keeping the\ndiscussion engaging.\n- Body language:\nObserve nonverbal cues\nsuch as leaning in, mirroring posture,\nfacial expressions, and gestures that convey\nenthusiasm, curiosity, or openness.\nThese\nsignals suggest a deeper level of interest\nbeyond mere politeness.\n2.\nContent and depth:\n- Topic selection:\nDoes the person show genuine\ncuriosity or passion about the subject matter,\nasking relevant questions, sharing insights, or\nexpressing opinions?\nIf they seem invested in\nexploring the topic’s nuances, it indicates a\ngenuine interest beyond mere politeness.\n- Level of detail:\nDoes the person provide\nspecific examples, personal anecdotes, or\nelaborate on their thoughts, indicating a\ndeeper level of engagement and investment in\nthe conversation?\n3.\nFollow-up actions:\n- Continued communication:\nDoes the person\ninitiate further conversations, send follow-up\nmessages, or suggest meeting up outside the\ninitial interaction?\nSuch actions suggest a\ngenuine desire to maintain the connection and\nexplore further topics, indicating a genuine\ninterest beyond mere politeness.\n- Personalization:\nDoes the person remember\ndetails discussed earlier, reference shared\ninterests, or show interest in learning more\nabout your life or experiences?\nPersonalized\ninteractions indicate a genuine interest in\nbuilding a relationship beyond mere politeness.\nOverall, genuine interest in a conversation\ninvolves active participation, enthusiasm,\ndepth, and follow-up actions that demonstrate\na genuine desire to connect and engage beyond\nmere politeness.\n27\n",
  "28": "Figure 11: A case study on AlpacaEval 2 demonstrates that Llama-3-Instruct, trained with SimPO,\nprovides a better formatted and more detailed answer than both Llama-3-Base, also trained with\nSimPO, and the baseline model GPT-4-1106-Preview. This illustrates how the instruction setting\ntypically outperforms the base setting.\nInput →\nQ: what language does argentina people\nspeak\nLlama-3-Base SimPO →\nThe official language spoken in Argentina\nis Spanish, specifically Argentine\nSpanish, which shares similarities with\nother varieties of Spanish spoken in\nneighboring countries like Uruguay, Chile,\nand Paraguay.\nLlama-3-Instruct SimPO →\nIn Argentina, the official language is\nSpanish.\nSpecifically, the dialect spoken\nin Argentina is known as Rioplatense\nSpanish, which is a variation of Spanish\nwith some unique characteristics and\ninfluences from Italian, German, and other\nimmigrant languages.\nRioplatense Spanish is distinguished by:\n1.\nVoseo:\nInstead of using tú (you)\nas the informal singular pronoun,\nArgentines use vos and its associated verb\nconjugations.\n2.\nLunfardo:\nA slang vocabulary derived\nfrom Italian, Corsican, and other immigrant\nlanguages, which has been incorporated into\neveryday speech.\n3.\nAccent and pronunciation:\nArgentines\ntend to pronounce the \"c\" and \"z\" sounds\nwith a soft \"s\" or \"sh\" sound, rather than\na hard \"th\" sound like in standard Spanish.\nOverall, Argentine Spanish is known for its\ndistinctive flavor and is widely understood\nthroughout the Spanish-speaking world.\nHowever, it’s worth noting that there may\nbe regional variations within Argentina,\nand some dialects may differ slightly from\nthe standard Rioplatense variety.\ngpt4-turbo-1106-preview →\nThe primary language spoken in Argentina\nis Spanish.\nSpecifically, the variety\nof Spanish spoken there is known as\nRioplatense Spanish, which is characterized\nby its unique accent and the use of the\npronoun \"vos\" instead of \"tú\" for the\nsecond person singular.\nAdditionally,\nthere are various indigenous languages\nspoken in different regions of the country,\nas well as immigrant languages such as\nItalian, German, and English, but these\nare much less widespread than Spanish.\n28\n",
  "29": "H\nLlama-3-Instruct v0.2 (Jul 7, 2024)\nTable 12: Results of Llama-3-Instruct (8B) setting, utilizing preference labels annotated by a stronger\nreward model (ArmoRM [84], we term it as version 0.2).\nMethod\nLlama-3-Instruct (8B)\nAlpacaEval 2\nArena-Hard\nMT-Bench\nLC (%)\nWR (%)\nLength\nWR (%)\nLength\nGPT-4 Turbo\nGPT-4\nSFT\n26.0\n25.3\n1920\n22.3\n596\n6.9\n8.1\nSimPO v0.1\n44.7\n40.5\n1825\n33.8\n504\n7.0\n8.0\nRRHF [91]\n37.9\n31.6\n1700\n28.8\n467\n7.1\n8.2\nSLiC-HF [96]\n33.9\n32.5\n1938\n29.3\n599\n6.9\n8.1\nDPO [66]\n48.2\n47.5\n2000\n35.2\n609\n7.0\n8.2\nIPO [6]\n46.8\n42.4\n1830\n36.6\n527\n7.2\n8.2\nCPO [88]\n34.1\n36.4\n2086\n30.9\n604\n7.2\n8.2\nKTO [29]\n34.1\n32.1\n1878\n27.3\n541\n7.2\n8.2\nORPO [42]\n38.1\n33.8\n1803\n28.2\n520\n7.2\n8.3\nR-DPO [64]\n48.0\n45.8\n1933\n35.1\n608\n7.0\n8.2\nSimPO v0.2\n53.7\n47.5\n1777\n36.5\n530\n7.0\n8.0\nTable 13: Downstream task evaluation results of tasks on the huggingface open leaderboard.\nMMLU (5)\nARC (25)\nHellaSwag (10)\nTruthfulQA (0)\nWinograd (5)\nGSM8K (5)\nAverage\nLlama-3-Instruct\nSFT\n67.06\n61.01\n78.57\n51.66\n74.35\n68.69\n66.89\nRRHF\n67.20\n61.52\n79.54\n53.76\n74.19\n66.11\n67.05\nSLiC-HF\n66.41\n61.26\n78.80\n53.23\n76.16\n66.57\n67.07\nDPO\n66.88\n63.99\n80.78\n59.01\n74.66\n49.81\n65.86\nIPO\n66.52\n61.95\n77.90\n54.64\n73.09\n58.23\n65.39\nCPO\n67.05\n62.29\n78.73\n54.01\n73.72\n67.40\n67.20\nKTO\n66.38\n63.57\n79.51\n58.15\n73.40\n57.01\n66.34\nORPO\n66.41\n61.01\n79.38\n54.37\n75.77\n64.59\n66.92\nR-DPO\n66.74\n64.33\n80.97\n60.32\n74.82\n43.90\n65.18\nSimPO\n65.63\n62.80\n78.33\n60.70\n73.32\n50.72\n65.25\nLlama-3-Instruct v0.2\nSFT\n67.06\n61.01\n78.57\n51.66\n74.35\n68.69\n66.89\nRRHF\n66.60\n63.74\n80.98\n59.40\n76.32\n58.68\n67.62\nSLiC-HF\n66.91\n61.77\n79.17\n56.36\n76.40\n68.23\n68.14\nDPO\n67.33\n64.08\n80.08\n56.33\n75.61\n54.51\n66.32\nIPO\n67.32\n63.23\n78.71\n58.12\n74.51\n56.33\n66.37\nCPO\n66.86\n62.80\n79.10\n55.62\n73.88\n67.78\n67.67\nKTO\n67.25\n63.57\n79.66\n55.56\n74.98\n66.41\n67.91\nORPO\n66.78\n63.40\n80.09\n57.52\n76.72\n66.72\n68.54\nR-DPO\n67.28\n64.51\n80.22\n56.44\n75.61\n52.99\n66.17\nSimPO\n66.51\n66.64\n78.97\n63.86\n74.74\n55.65\n67.73\nSimPO w/ SFT\n66.74\n63.82\n78.82\n60.52\n73.72\n64.06\n67.95\nIn this section, we update the Llama-3-Instruct setting, primarily by utilizing a stronger reward model\nto annotate our generated preference dataset.\nEnhanced reward model yields significantly better results.\nIn our previous version, we use\nPairRM [45] as our reward model to rank generated candidate responses. The results, presented in\nTable 12, show that switching the reward model from PairRM [45] to ArmoRM [84] for ranking\nthe data markedly improves model performance. This underscores the importance of a high-quality\npreference optimization dataset for enhancing performance. Notably, SimPO has achieved a 53.7 LC\nwin rate on AlpacaEval 2 and 36.5 on Arena-Hard, surpassing the previous version by 9.0 and 2.7\npoints, respectively.\nWe use the following hyperparameters for SimPO under the Llama-3 Instruct v0.2 setting: β = 10\nand γ = 3. The other hyperparameters (e.g., learning rate, batch size, max sequence lengths) are kept\nthe same as the original Llama-3-8B-Instruct setting.\n29\n",
  "30": "Strong SFT model and high-quality policy data diminish algorithm differences.\nWith a strong\nSFT model like Llama-3-8B-Instruct, and as the preference optimization data quality improves, the\ndifferences between algorithms become less pronounced. For instance, DPO achieved a similar win\nrate as SimPO in terms of raw win rate, and DPO, IPO, and R-DPO all exhibited comparable raw\nwin rates on Arena-Hard. However, SimPO maintains an advantage by producing shorter sequences,\nresulting in a significantly better LC win rate on AlpacaEval 2.\nTable 14: AlpacaEval 2 performance of\nSimPO and SimPO with an SFT loss.\nMethod\nLC (%)\nWR (%)\nSimPO v0.2\n53.7\n47.5\nw/ SFT\n41.4\n36.5\nStronger downstream task performance.\nThe v0.2 ver-\nsion also shows improved performance in downstream tasks\nacross various objectives. However, DPO, IPO, R-DPO,\nand SimPO continue to experience a decline in reasoning-\nintensive domains such as GSM8K. In contrast, objectives\nthat include an SFT component maintain their performance\nin mathematical tasks.\nIncorporating SFT regularization in SimPO.\nSeveral reference-free algorithms, including\nRRHF [91], SLiC-HF [96], CPO [88], and ORPO [42], employ SFT regularization in their ob-\njectives. SFT regularization can be an effective method to prevent reward hacking, ensuring that the\nsolution maintains low loss without resulting in degraded generations. We also experiment with the\nintegration of an SFT loss in SimPO, yielding the following objective:\nLSimPO w/ SFT(πθ) = −E(x,yw,yl)∼D\n\u0014\nlog σ\n\u0012 β\n|yw| log πθ(yw|x) −β\n|yl| log πθ(yl|x) −γ\n\u0013\n+ λ log πθ(yw|x)\n\u0015\n.\nAs shown in Table 14, the addition of the SFT regularization leads to a decrease in performance on\nAlpacaEval 2. However, we note that SFT regularization provides substantial benefits to certain tasks\nsuch as GSM8K, as shown in Table 12. These contrasting results suggest that the impact of SFT in\npreference optimization may vary depending on the training setup and the nature of the task. Further\ncomprehensive studies on this topic are left for future research.\nI\nApplying Length Normalization and Target Reward Margin to DPO (Jul 7,\n2024)\nSince the release of the paper, we have had inquiries from researchers about whether the key design\nelements of SimPO—length normalization and target reward margin—could benefit DPO. By doing\nso, we will derive the following two objectives:\nLDPO w/ LN(πθ; πref) = −E(x,yw,yl)∼D\n\u0014\nlog σ\n\u0012 β\n|yw| log πθ(yw | x)\nπref(yw | x) −β\n|yl| log πθ(yl | x)\nπref(yl | x)\n\u0013\u0015\n.\nLDPO w/ γ(πθ; πref) = −E(x,yw,yl)∼D\n\u0014\nlog σ\n\u0012\nβ log πθ(yw | x)\nπref(yw | x) −β log πθ(yl | x)\nπref(yl | x) −γ\n\u0013\u0015\n.\nAn intuitive understanding of how length normalization could benefit DPO is that, despite DPO’s\nreward design being implicitly normalized by the reference model, the policy model might still exploit\nlength bias from the data, resulting in a disproportionately high probability for longer sequences.\nApplying length normalization could help mitigate this effect.\nWe train models with the objectives mentioned above and compare their performance to that of DPO\nand SimPO, as shown in Table 15.\nThe results indicate that, unlike SimPO, length normalization and target reward margin do not\nconsistently benefit DPO. Specifically, length normalization significantly improves DPO performance\nonly in the Mistral-Base setting, where the preference optimization dataset shows a strong length\nbias. However, it does not provide a benefit in the Mistral-Instruct setting, where the lengths of\nwinning and losing responses are comparable. This is likely because DPO already includes an implicit\ninstance-wise target reward margin via the reference model, as shown in the derivation below.\nLDPO = log σ\n\u0012\nβ log πθ(yw | x)\nπref(yw | x) −β log πθ(yl | x)\nπref(yl | x)\n\u0013\n= log σ\n\u0012\nβ log πθ(yw | x) −β log πθ(yl | x) −\n\u0000β log πref(yw | x) −β log πref(yl | x)\n\u0001\n|\n{z\n}\n=γref\n\u0013\n.\n30\n",
  "31": "Table 15: Applying length normalization (LN) and target reward margin (γ) to DPO.\nMethod\nMistral-Base (7B) Setting\nMistral-Instruct (7B) Setting\nAlpacaEval 2 Arena-Hard\nMT-Bench\nAlpacaEval 2 Arena-Hard\nMT-Bench\nLC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4 LC (%) WR (%)\nWR (%)\nGPT-4 Turbo GPT-4\nSimPO\n21.5\n20.8\n16.6\n6.0\n7.3\n32.1\n34.8\n21.0\n6.6\n7.6\nDPO\n15.1\n12.5\n10.4\n5.9\n7.3\n26.8\n24.9\n16.3\n6.3\n7.6\nw/ LN\n21.0\n17.7\n15.2\n5.9\n7.2\n21.7\n20.9\n15.6\n6.4\n7.7\nw/ γ\n15.2\n12.1\n10.3\n5.7\n7.3\n23.0\n24.6\n14.7\n6.3\n7.6\nJ\nApplying SimPO to Gemma 2 Models (Sept 16, 2024)\nPerformance degradation on other benchmarks for Llama-3-SimPO checkpoints.\nAfter releas-\ning the Llama-3-SimPO checkpoints, we received extensive feedback about performance degradation\non benchmarks measuring specific capabilities, such as MMLU and GSM8K. To investigate this\nissue, we continued training the Llama-3-8B-Instruct model with different learning rates, as reported\nin Table 16. We find that using a higher learning rate results in a stronger model in chat-oriented\nbenchmarks, at the cost of catastrophic forgetting on GSM8K and MMLU.17 With a smaller learning\nrate, the model’s performance on chat benchmarks is slightly worse, but its performance on GSM8K\nand MMLU is better retained. This demonstrates a trade-off between chat-oriented benchmarks and\nother benchmarks when continuing training from a strong instruction-tuned model.\nTable 16: Results on AlpacaEval 2, ZeroEval GSM, and ZeroEval MMLU when continuing training\nfrom Llama-3-8B-Instruct with different learning rates. * indicates the released checkpoint.\nModel\nAlpacaEval 2\nZeroEval\nZeroEval\nLC (%)\nGSM (%)\nMMLU (%)\nLlama-3-8B-Instruct\n26.0\n78.5\n61.7\nSimPO (lr=4e-7)\n38.8\n77.9\n62.6\nSimPO (lr=5e-7)\n44.6\n77.0\n62.3\nSimPO (lr=1e-6)*\n53.7\n57.4\n54.9\nApplying SimPO to Gemma 2 models presents a different trend.\nWe evaluate SimPO using\nGoogle’s recently released Gemma-2-9B-it model [77], which represents a strong open-source model.\nFor training data, we generate up to 5 responses per prompt from the UltraFeedback dataset [23] and\nuse the ArmoRM model [84] to annotate preferences between responses. We compare our SimPO\nagainst a DPO-trained variant, both fine-tuned from the Gemma-2-9B-it base model. As shown in\nAppendix J, SimPO demonstrates superior performance on chat benchmarks like AlpacaEval 2 and\nArena-Hard while maintaining the model’s original zero-shot capabilities on tasks like GSM8K and\nMMLU. Notably, we find that varying the learning rate during fine-tuning has minimal impact on the\nmodel’s performance. These results suggest an underlying property difference between the Llama-3\ncheckpoints and the Gemma 2 checkpoints, and might be worth further investigation.\nGemma-2-9B-it-SimPO significantly improved the ranking of the Gemma-2-9B-it model on\nChatbot Arena.\nDuring the development stage, we relied solely on automated metrics to evaluate\nthe model’s performance. To determine if these metrics aligned with real user preferences, we\nsubmitted our best-performing model, Gemma-2-9B-it-SimPO, to the Chatbot Arena leaderboard\nhosted by LMSYS [17]. We find that our model improved the original Gemma-2-9B-it ranking\nfrom 36th to 25th, making the SimPO variant the top-ranked <10B model on the Chatbot Arena\nleaderboard based on real user votes as of September 16th, 2024.\n17We evaluate the zero-shot performance of the models on GSM8K and MMLU using the ZeroEval repository\nwhich adopts a unified setup.\n31\n",
  "32": "Table 17: Benchmark performance of Gemma-2-9B trained with DPO and SimPO on UltraFeedback\n(responses regenerated with Gemma-2-9B-it, following the same dataset construction process as\nLlama-3-Instruct (8B) described in Section 3). SimPO results in better instruction following perfor-\nmance than DPO without degrading math abilities (GSM) or general knowledge (MMLU) of the\noriginal model. * indicates the released checkpoint.\nModel\nInstruction Following\nCapabilities\nAlpacaEval 2 LC\nArena-Hard\nZeroEval\nZeroEval\n(%)\n(%)\nGSM (%)\nMMLU (%)\nGemma-2-9B-it\n51.1\n40.8\n87.4\n72.7\nGemma-2-9B-DPO\n67.8\n58.9\n88.5\n72.2\nGemma-2-9B-SimPO (lr=6e-7)\n71.7\n58.3\n88.3\n72.2\nGemma-2-9B-SimPO (lr=8e-7)*\n72.4\n59.1\n88.0\n72.2\nGemma-2-9B-SimPO (lr=1e-6)\n71.0\n58.3\n87.4\n71.5\n32\n"
}