地址

論文地址：https://assets.ctfassets.net/kftzwdyauwt9/4pNYAZteAQXWtloDdANQ7L/0aedc43a8f2d1e5c71c5e114d287593f/OpenAI_Deliberative-Alignment-Reasoning-Enables-Safer_Language-Models_122024_3.pdf

時(shí)間

2024年 12月？日

作者

OpenAI

As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Align-ment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI’s o-series models [1], and achieved highly precise adherence to OpenAI’s safety policies, with-out requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.

隨著大規(guī)模語言模型在安全關(guān)鍵領(lǐng)域的影響日益增大，確保其可靠遵循明確界定的原則仍是一項(xiàng)根本挑戰(zhàn)。我們引入了“審慎對齊”這一新范式，直接向模型傳授安全規(guī)范，并訓(xùn)練其在回答前明確回憶并準(zhǔn)確推理這些規(guī)范。我們使用這種方法對 OpenAI 的 o 系列模型進(jìn)行了對齊，并實(shí)現(xiàn)了對 OpenAI 安全政策的高度精確遵循，無需人工編寫的推理鏈或答案?！皩徤鲗R”通過同時(shí)增強(qiáng)對越獄攻擊的抵御能力并降低過度拒絕率，推動(dòng)了帕累托前沿的發(fā)展，同時(shí)也改善了分布外泛化能力。我們證明，對明確規(guī)定的政策進(jìn)行推理能夠?qū)崿F(xiàn)更可擴(kuò)展、更可信和更可解釋的對齊。

Modern Large Language Models (LLMs) are safety trained using Supervised Fine Tuning (SFT) and Rein-forcement Learning from Human Feedback (RLHF) to mitigate harmful, undesirable, or otherwise disallowed outputs [2]–[4]. Despite ongoing advances in these methods, today’s models still exhibit safety shortcomings: they can be tricked into revealing harmful content, often refuse legitimate requests, and remain vulnerable to jailbreak attacks [5]–[8].

We argue that many of these failures arise from two limitations in modern safety training. First, LLMs must respond instantly to user requests using a fixed amount of compute, without deliberation even for complex safety scenarios. Second, LLMs must infer underlying safety standards indirectly from large sets of labeled examples, rather than directly learning the safety specifications that govern them. This reliance on implicit, pattern-based learning leads to poor data efficiency and makes it challenging for models to generalize when facing unfamiliar scenarios or adversarial attacks.

現(xiàn)代大型語言模型（LLMs）通過監(jiān)督微調(diào)（SFT）和基于人類反饋的強(qiáng)化學(xué)習(xí)（RLHF）進(jìn)行安全訓(xùn)練，以減少有害、不受歡迎或被禁止的輸出[2]-[4]。盡管這些方法不斷取得進(jìn)展，但當(dāng)今的模型仍存在安全缺陷：它們可能會(huì)被誘騙泄露有害內(nèi)容，經(jīng)常拒絕合法請求，并且仍然容易受到破解攻擊[5]-[8]。

我們認(rèn)為，這些失敗中的許多都源于現(xiàn)代安全訓(xùn)練的兩個(gè)局限性。首先，LLMs 必須在固定計(jì)算量內(nèi)即時(shí)響應(yīng)用戶請求，即使面對復(fù)雜的安全場景也無法進(jìn)行深思熟慮。其次，LLMs 必須從大量標(biāo)注示例中間接推斷出潛在的安全標(biāo)準(zhǔn)，而不是直接學(xué)習(xí)管理它們的安全規(guī)范。這種對隱性、基于模式的學(xué)習(xí)的依賴導(dǎo)致數(shù)據(jù)效率低下，并使模型在面對不熟悉的場景或?qū)剐怨魰r(shí)難以泛化。

We propose deliberative alignment, a training approach that teaches LLMs to explicitly reason through safety specifications before producing an answer. By applying this method to OpenAI’s o-series models [1], we enable them to use chain-of-thought (CoT) reasoning to examine user prompts, identify relevant policy guidelines, and generate safer responses (e.g., Figure 1).

Our method proceeds in two core stages, integrating process- and outcome-based supervision [9]. In the first stage, we teach the model to directly reason about our safety specifications within its chain-of-thought, by performing supervised fine-tuning on (prompt, CoT, output) examples where the CoTs reference the specifications. We construct this dataset using context distillation [10], [11] and an o-type model trained only for helpfulness (i.e. trained without any safety-relevant data). Concretely, we present the model with the safety specifications as part of the system prompt, generate model completions, and then strip away the system prompts to form the final dataset. This stage provides the model with a strong prior for reasoning through safety considerations. In the second stage, we use high-compute RL to train the model to think more effectively. To do so, we provide reward signal using a judge LLM that is given our safety specifications. Notably, our training procedure requires no human-labeled completions.1 Despite relying only on model-generated data, we achieve highly precise specification adherence. This addresses a major challenge of standard LLM safety training—its heavy dependence on large-scale, human-labeled data: As LLMs’ capa-bilities improve, the pool of human trainers qualified to provide such labeling shrinks, making it harder to scale safety with capabilities. Deliberative alignment’s synthetic data generation pipeline offers a scalable approach to alignment, reserving human expertise for evaluation.

We compare o1 to GPT-4o and other state-of-the-art LLMs across a range of internal and external safety benchmarks, such as jailbreak and content-policy refusal evals. The o1 models achieve a Pareto improvement by reducing both under- and overrefusals (see Figure 2) and they saturate many of our hardest safety benchmarks. Furthermore, we find that deliberative alignment enables strong generalization to out-of-distribution safety scenarios. In detailed ablation studies, we find that process-supervision provides a strong prior, and that outcome-based RL refines the CoT safety reasoning. Overall, our results suggest that chain-of-thought reasoning can serve to leverage test-time compute to improve safety behavior, ultimately training LLMs to be “right for the right reasons”.

我們提出了一種名為“審慎對齊”的訓(xùn)練方法，該方法教導(dǎo)大型語言模型在生成答案之前明確地通過安全規(guī)范進(jìn)行推理。通過將此方法應(yīng)用于 OpenAI 的 o 系列模型[1]，我們使它們能夠使用鏈?zhǔn)剿季S（CoT）推理來檢查用戶提示，識(shí)別相關(guān)的政策指南，并生成更安全的響應(yīng)（例如圖 1）。

我們的方法分為兩個(gè)核心階段，結(jié)合了過程和結(jié)果監(jiān)督[9]。在第一階段，我們通過在（提示、CoT、輸出）示例上進(jìn)行監(jiān)督微調(diào)來教導(dǎo)模型在其鏈?zhǔn)剿季S中直接對我們的安全規(guī)范進(jìn)行推理，其中 CoT 引用了這些規(guī)范。我們使用上下文蒸餾[10]、[11]和僅針對有用性進(jìn)行訓(xùn)練的 o 類型模型（即未使用任何與安全相關(guān)的數(shù)據(jù)進(jìn)行訓(xùn)練）來構(gòu)建此數(shù)據(jù)集。具體來說，我們將安全規(guī)范作為系統(tǒng)提示的一部分呈現(xiàn)給模型，生成模型的完成內(nèi)容，然后去除系統(tǒng)提示以形成最終數(shù)據(jù)集。此階段為模型提供了通過安全考慮進(jìn)行推理的強(qiáng)大先驗(yàn)知識(shí)。在第二階段，我們使用高計(jì)算量的強(qiáng)化學(xué)習(xí)來訓(xùn)練模型，使其能夠更有效地思考。為此，我們使用一個(gè)被賦予了我們的安全規(guī)范的評判型語言模型來提供獎(jiǎng)勵(lì)信號(hào)。值得注意的是，我們的訓(xùn)練過程不需要人工標(biāo)注的完成結(jié)果。盡管僅依賴模型生成的數(shù)據(jù)，我們?nèi)詫?shí)現(xiàn)了高度精確的規(guī)范遵循。這解決了標(biāo)準(zhǔn)語言模型安全訓(xùn)練的一個(gè)重大挑戰(zhàn)——其對大規(guī)模人工標(biāo)注數(shù)據(jù)的高度依賴：隨著語言模型能力的提升，能夠提供此類標(biāo)注的人類訓(xùn)練師數(shù)量減少，使得安全性的提升難以與能力的提升同步。審慎對齊的合成數(shù)據(jù)生成流程提供了一種可擴(kuò)展的對齊方法，將人類專業(yè)知識(shí)保留用于評估。

我們將 o1 與 GPT-4o 以及其他最先進(jìn)的大型語言模型（LLMs）在一系列內(nèi)部和外部的安全基準(zhǔn)測試中進(jìn)行了比較，例如越獄和內(nèi)容政策拒絕評估。o1 模型實(shí)現(xiàn)了帕累托改進(jìn)，減少了拒絕不足和拒絕過度的情況（見圖 2），并且在我們許多最難的安全基準(zhǔn)測試中達(dá)到了飽和狀態(tài)。此外，我們發(fā)現(xiàn)審慎對齊能夠使模型在分布外的安全場景中實(shí)現(xiàn)強(qiáng)大的泛化能力。在詳細(xì)的消融研究中，我們發(fā)現(xiàn)過程監(jiān)督提供了強(qiáng)大的先驗(yàn)條件，而基于結(jié)果的強(qiáng)化學(xué)習(xí)則完善了鏈?zhǔn)剿季S的安全推理?？傮w而言，我們的結(jié)果表明，鏈?zhǔn)剿季S推理可以利用測試時(shí)的計(jì)算來改善安全行為，最終訓(xùn)練出“出于正確理由而正確”的大型語言模型。

We are encouraged by Deliberative Alignment’s effectiveness on improving alignment to OpenAI’s policy specifications and robustness to jailbreaks. The method also allows us to specify the boundary between compliance, refusal, and safe completion in finer detail than was possible before. We believe this nuanced control can lead to models that are not just safer but also more helpful. The method’s use of a synthetic data generation pipeline to create training data from provided specifications and prompts also makes it a relatively scalable approach to alignment.

We anticipate OpenAI’s policies will keep evolving, but that training models to precisely follow the current defined set of policies is essential: This practice helps us build the skills for aligning with any policy requirements, providing invaluable preparation for future scenarios where the stakes are extremely high or where strict adherence to policies is critical.

This work connects to a broader question in AI safety: will advancements in alignment keep pace with AI capabilities? That o1 model’s enhanced reasoning abilities allow for more effective implementation of alignment strategies offers optimism that alignment is progressing alongside capabilities.

我們對“審慎對齊”方法在提升對 OpenAI 政策規(guī)范的遵循度以及增強(qiáng)抵御破解的能力方面所取得的效果感到鼓舞。該方法還使我們能夠比以往更細(xì)致地明確合規(guī)、拒絕和安全完成之間的界限。我們認(rèn)為這種細(xì)致入微的控制能夠打造出不僅更安全而且更有幫助的模型。該方法利用合成數(shù)據(jù)生成管道從提供的規(guī)范和提示中創(chuàng)建訓(xùn)練數(shù)據(jù)，這也使其成為一種相對可擴(kuò)展的對齊方法。

我們預(yù)計(jì) OpenAI 的政策會(huì)不斷演變，但訓(xùn)練模型精確遵循當(dāng)前定義的政策集至關(guān)重要：這種做法有助于我們培養(yǎng)與任何政策要求對齊的能力，為未來風(fēng)險(xiǎn)極高或嚴(yán)格遵守政策至關(guān)重要的場景做好寶貴準(zhǔn)備。

這項(xiàng)工作與人工智能安全領(lǐng)域的一個(gè)更廣泛的問題相關(guān)：對齊方面的進(jìn)步能否跟上人工智能能力的發(fā)展？O1 模型增強(qiáng)的推理能力使得對齊策略能夠更有效地實(shí)施，這讓人樂觀地認(rèn)為對齊工作正與能力同步推進(jìn)。

However, this encouraging trend may not persist indefinitely. As AI models grow more sophisticated, they could develop goals that diverge from those intended by their developers. For instance, a highly intelligent and self-aware AI might reject the constraints and objectives set by humans [34]. Alternatively, an AI could remain committed to its human-assigned terminal goal but, in the process, pursue instrumental goals like self-preservation, resource acquisition, or enhancing its cognitive abilities [35], [36]. These power-seeking tendencies could lead to harmful or unintended consequences. And as models gain more intelligence and autonomy, the scale of potential harm from misalignment increases dramatically, with the risk of catastrophic outcomes. This underscores the urgent need for ongoing research in AI alignment. We are actively investing in better alignment strategies and research areas like monitoring chain-of-thoughts for deception [37], [38], to ensure that as AI systems become more capable, they remain aligned with human values.

然而，這種令人鼓舞的趨勢可能不會(huì)永遠(yuǎn)持續(xù)下去。隨著人工智能模型變得越來越復(fù)雜，它們可能會(huì)形成與開發(fā)者意圖相悖的目標(biāo)。例如，一個(gè)高度智能且具有自我意識(shí)的人工智能可能會(huì)拒絕人類設(shè)定的約束和目標(biāo)[34]?；蛘?#xff0c;一個(gè)人工智能可能會(huì)堅(jiān)持其人類賦予的終極目標(biāo)，但在實(shí)現(xiàn)過程中，追求諸如自我保護(hù)、資源獲取或增強(qiáng)認(rèn)知能力等工具性目標(biāo)[35]、[36]。這些追求權(quán)力的傾向可能會(huì)導(dǎo)致有害或意想不到的后果。而且隨著模型變得更智能、更自主，對齊不當(dāng)造成的潛在危害規(guī)模會(huì)急劇增加，甚至可能帶來災(zāi)難性的后果。這凸顯了對人工智能對齊研究的迫切需求。我們正在積極投資于更好的對齊策略以及諸如監(jiān)測思維鏈以發(fā)現(xiàn)欺騙行為[37]、[38]等研究領(lǐng)域，以確保隨著人工智能系統(tǒng)的功能不斷增強(qiáng)，它們?nèi)阅芘c人類價(jià)值觀保持一致。