自我发现工作流¶

本 Notebook 展示了如何实现 SELF-DISCOVER。

对于给定任务，它包含两个阶段

1. 阶段1

   a. 选择：选择推理模块的子集。

   b. 调整：将选定的推理模块调整以适应任务。

   c. 实现：为任务提供推理结构。

2. 阶段2：使用为任务生成的推理结构来生成答案。

此实现的灵感来源于代码库

In [ ]

%pip install -U llama-index

%pip install -U llama-index

In [ ]

import os

os.environ["OPENAI_API_KEY"] = "<Your OpenAI API Key>"

import os os.environ["OPENAI_API_KEY"] = ""

由于工作流是异步优先的，这在 Notebook 中运行良好。如果你在自己的代码中运行，如果还没有运行异步事件循环，你会希望使用 asyncio.run() 来启动它。

async def main():
    <async code>

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

设置¶

设置推理模块和提示词模板。

In [ ]

from llama_index.core.prompts import PromptTemplate

_REASONING_MODULES = [
    "1. How could I devise an experiment to help solve that problem?",
    "2. Make a list of ideas for solving this problem, and apply them one by one to the problem to see if any progress can be made.",
    "3. How could I measure progress on this problem?",
    "4. How can I simplify the problem so that it is easier to solve?",
    "5. What are the key assumptions underlying this problem?",
    "6. What are the potential risks and drawbacks of each solution?",
    "7. What are the alternative perspectives or viewpoints on this problem?",
    "8. What are the long-term implications of this problem and its solutions?",
    "9. How can I break down this problem into smaller, more manageable parts?",
    "10. Critical Thinking: This style involves analyzing the problem from different perspectives, questioning assumptions, and evaluating the evidence or information available. It focuses on logical reasoning, evidence-based decision-making, and identifying potential biases or flaws in thinking.",
    "11. Try creative thinking, generate innovative and out-of-the-box ideas to solve the problem. Explore unconventional solutions, thinking beyond traditional boundaries, and encouraging imagination and originality.",
    "12. Seek input and collaboration from others to solve the problem. Emphasize teamwork, open communication, and leveraging the diverse perspectives and expertise of a group to come up with effective solutions.",
    "13. Use systems thinking: Consider the problem as part of a larger system and understanding the interconnectedness of various elements. Focuses on identifying the underlying causes, feedback loops, and interdependencies that influence the problem, and developing holistic solutions that address the system as a whole.",
    "14. Use Risk Analysis: Evaluate potential risks, uncertainties, and tradeoffs associated with different solutions or approaches to a problem. Emphasize assessing the potential consequences and likelihood of success or failure, and making informed decisions based on a balanced analysis of risks and benefits.",
    "15. Use Reflective Thinking: Step back from the problem, take the time for introspection and self-reflection. Examine personal biases, assumptions, and mental models that may influence problem-solving, and being open to learning from past experiences to improve future approaches.",
    "16. What is the core issue or problem that needs to be addressed?",
    "17. What are the underlying causes or factors contributing to the problem?",
    "18. Are there any potential solutions or strategies that have been tried before? If yes, what were the outcomes and lessons learned?",
    "19. What are the potential obstacles or challenges that might arise in solving this problem?",
    "20. Are there any relevant data or information that can provide insights into the problem? If yes, what data sources are available, and how can they be analyzed?",
    "21. Are there any stakeholders or individuals who are directly affected by the problem? What are their perspectives and needs?",
    "22. What resources (financial, human, technological, etc.) are needed to tackle the problem effectively?",
    "23. How can progress or success in solving the problem be measured or evaluated?",
    "24. What indicators or metrics can be used?",
    "25. Is the problem a technical or practical one that requires a specific expertise or skill set? Or is it more of a conceptual or theoretical problem?",
    "26. Does the problem involve a physical constraint, such as limited resources, infrastructure, or space?",
    "27. Is the problem related to human behavior, such as a social, cultural, or psychological issue?",
    "28. Does the problem involve decision-making or planning, where choices need to be made under uncertainty or with competing objectives?",
    "29. Is the problem an analytical one that requires data analysis, modeling, or optimization techniques?",
    "30. Is the problem a design challenge that requires creative solutions and innovation?",
    "31. Does the problem require addressing systemic or structural issues rather than just individual instances?",
    "32. Is the problem time-sensitive or urgent, requiring immediate attention and action?",
    "33. What kinds of solution typically are produced for this kind of problem specification?",
    "34. Given the problem specification and the current best solution, have a guess about other possible solutions."
    "35. Let’s imagine the current best solution is totally wrong, what other ways are there to think about the problem specification?"
    "36. What is the best way to modify this current best solution, given what you know about these kinds of problem specification?"
    "37. Ignoring the current best solution, create an entirely new solution to the problem."
    "38. Let’s think step by step ."
    "39. Let’s make a step by step plan and implement it with good notation and explanation.",
]

_REASONING_MODULES = "\n".join(_REASONING_MODULES)

SELECT_PRMOPT_TEMPLATE = PromptTemplate(
    "Given the task: {task}, which of the following reasoning modules are relevant? Do not elaborate on why.\n\n {reasoning_modules}"
)

ADAPT_PROMPT_TEMPLATE = PromptTemplate(
    "Without working out the full solution, adapt the following reasoning modules to be specific to our task:\n{selected_modules}\n\nOur task:\n{task}"
)

IMPLEMENT_PROMPT_TEMPLATE = PromptTemplate(
    "Without working out the full solution, create an actionable reasoning structure for the task using these adapted reasoning modules:\n{adapted_modules}\n\nTask Description:\n{task}"
)

REASONING_PROMPT_TEMPLATE = PromptTemplate(
    "Using the following reasoning structure: {reasoning_structure}\n\nSolve this task, providing your final answer: {task}"
)

from llama_index.core.prompts import PromptTemplate _REASONING_MODULES = [ "1. 如何设计一个实验来帮助解决这个问题？", "2. 列出解决这个问题的方法清单，并逐个应用到问题上，看看是否能取得进展。", "3. 如何衡量在这个问题上的进展？", "4. 如何简化问题使其更容易解决？", "5. 这个问题潜在的关键假设是什么？", "6. 每种解决方案的潜在风险和缺点是什么？", "7. 对于这个问题，有哪些替代的视角或观点？", "8. 这个问题及其解决方案的长期影响是什么？", "9. 我如何将这个问题分解成更小、更易于管理的部分？", "10. 批判性思维：这种风格涉及从不同角度分析问题，质疑假设，评估可用的证据或信息。它侧重于逻辑推理、基于证据的决策，以及识别思维中潜在的偏见或缺陷。", "11. 尝试创造性思维，生成创新和突破常规的想法来解决问题。探索非传统解决方案，超越传统界限思考，鼓励想象力和原创性。", "12. 寻求他人的意见和协作来解决问题。强调团队合作、开放沟通，以及利用团队中多样化的视角和专业知识来提出有效解决方案。", "13. 使用系统思维：将问题视为更大系统的一部分，并理解各种元素之间的相互关联。侧重于识别影响问题的潜在原因、反馈循环和相互依赖关系，并开发解决整个系统的整体解决方案。", "14. 使用风险分析：评估与问题不同解决方案或方法相关的潜在风险、不确定性和权衡。强调评估潜在后果以及成功或失败的可能性，并基于对风险和收益的平衡分析做出明智决策。", "15. 使用反思性思维：退后一步审视问题，花时间进行内省和自我反思。检查可能影响问题解决的个人偏见、假设和心智模型，并乐于从过去的经验中学习以改进未来的方法。", "16. 需要解决的核心问题是什么？", "17. 导致问题的潜在原因或因素是什么？", "18. 之前是否尝试过任何潜在的解决方案或策略？如果是，结果和吸取的教训是什么？", "19. 在解决这个问题时可能出现哪些潜在障碍或挑战？", "20. 是否有任何相关数据或信息可以提供对问题的见解？如果是，有哪些数据源可用，以及如何分析它们？", "21. 是否有任何直接受问题影响的利益相关者或个人？他们的视角和需求是什么？", "22. 有效解决问题需要哪些资源（财务、人力、技术等）？", "23. 如何衡量或评估解决问题的进展或成功？", "24. 可以使用哪些指标或衡量标准？", "25. 这个问题是技术性或实践性的，需要特定的专业知识或技能集吗？还是更多是概念性或理论性问题？", "26. 这个问题是否涉及物理约束，例如有限的资源、基础设施或空间？", "27. 这个问题是否与人类行为相关，例如社会、文化或心理问题？", "28. 这个问题是否涉及需要在不确定性下或具有竞争目标的情况下做出选择的决策或规划？", "29. 这个问题是分析性的，需要数据分析、建模或优化技术吗？", "30. 这个问题是需要创造性解决方案和创新的设计挑战吗？", "31. 这个问题是否需要解决系统性或结构性问题，而不仅仅是单个案例？", "32. 这个问题是否具有时效性或紧迫性，需要立即关注和采取行动？", "33. 对于这类问题规范通常会产生什么样的解决方案？", "34. 考虑到问题规范和当前最佳解决方案，猜测其他可能的解决方案。", "35. 假设当前最佳解决方案完全错误，还有哪些其他方式可以思考问题规范？", "36. 考虑到你对这类问题规范的了解，如何最佳地修改当前最佳解决方案？", "37. 忽略当前最佳解决方案，为问题创建一个全新的解决方案。", "38. 让我们一步一步思考。", "39. 让我们制定一个逐步计划，并以清晰的符号和解释来实现它。", ] _REASONING_MODULES = "\n".join(_REASONING_MODULES) SELECT_PRMOPT_TEMPLATE = PromptTemplate( "给定任务：{task}，以下哪些推理模块是相关的？不要详细阐述原因。\n\n {reasoning_modules}" ) ADAPT_PROMPT_TEMPLATE = PromptTemplate( "无需计算出完整解决方案，将以下推理模块调整为特定于我们的任务：\n{selected_modules}\n\n我们的任务：\n{task}" ) IMPLEMENT_PROMPT_TEMPLATE = PromptTemplate( "无需计算出完整解决方案，使用这些调整后的推理模块为任务创建一个可操作的推理结构：\n{adapted_modules}\n\n任务描述：\n{task}" ) REASONING_PROMPT_TEMPLATE = PromptTemplate( "使用以下推理结构：{reasoning_structure}\n\n解决此任务，并提供你的最终答案：{task}" )

设计工作流¶

SELF-DISCOVER 包含以下步骤

1. 选择给定推理模块的子集。
2. 基于任务精炼和调整给定推理模块的子集。
3. 根据调整后的推理模块为任务创建推理结构。
4. 根据推理结构为任务生成最终答案。

需要以下事件

1. GetModulesEvent：在检索到模块子集后触发。
2. RefineModulesEvent：在模块被精炼并调整以适应任务后触发。
3. ReasoningStructureEvent：在推理结构生成后触发。

In [ ]

from llama_index.core.workflow import Event


class GetModulesEvent(Event):
    """Event to get modules."""

    task: str
    modules: str


class RefineModulesEvent(Event):
    """Event to refine modules."""

    task: str
    refined_modules: str


class ReasoningStructureEvent(Event):
    """Event to create reasoning structure."""

    task: str
    reasoning_structure: str

from llama_index.core.workflow import ( Workflow, Context, StartEvent, StopEvent, step, ) from llama_index.core.llms import LLM class SelfDiscoverWorkflow(Workflow): """Self discover workflow.""" @step async def get_modules( self, ctx: Context, ev: StartEvent ) -> GetModulesEvent: """Get modules step.""" # get input data, store llm into ctx task = ev.get("task") llm: LLM = ev.get("llm") if task is None or llm is None: raise ValueError("'task' and 'llm' arguments are required.") await ctx.set("llm", llm) # format prompt and get result from LLM prompt = SELECT_PRMOPT_TEMPLATE.format( task=task, reasoning_modules=_REASONING_MODULES ) result = llm.complete(prompt) return GetModulesEvent(task=task, modules=str(result)) @step async def refine_modules( self, ctx: Context, ev: GetModulesEvent ) -> RefineModulesEvent: """Refine modules step.""" task = ev.task modules = ev.modules llm: LLM = await ctx.get("llm") # format prompt and get result prompt = ADAPT_PROMPT_TEMPLATE.format( task=task, selected_modules=modules ) result = llm.complete(prompt) return RefineModulesEvent(task=task, refined_modules=str(result)) @step async def create_reasoning_structure( self, ctx: Context, ev: RefineModulesEvent ) -> ReasoningStructureEvent: """Create reasoning structures step.""" task = ev.task refined_modules = ev.refined_modules llm: LLM = await ctx.get("llm") # format prompt, get result prompt = IMPLEMENT_PROMPT_TEMPLATE.format( task=task, adapted_modules=refined_modules ) result = llm.complete(prompt) return ReasoningStructureEvent( task=task, reasoning_structure=str(result) ) @step async def get_final_result( self, ctx: Context, ev: ReasoningStructureEvent ) -> StopEvent: """Gets final result from reasoning structure event.""" task = ev.task reasoning_structure = ev.reasoning_structure llm: LLM = await ctx.get("llm") # format prompt, get res prompt = REASONING_PROMPT_TEMPLATE.format( task=task, reasoning_structure=reasoning_structure ) result = llm.complete(prompt) return StopEvent(result=result)

下面是 SELF-DISCOVER 工作流的代码

In [ ]

from llama_index.core.workflow import (
    Workflow,
    Context,
    StartEvent,
    StopEvent,
    step,
)
from llama_index.core.llms import LLM


class SelfDiscoverWorkflow(Workflow):
    """Self discover workflow."""

    @step
    async def get_modules(
        self, ctx: Context, ev: StartEvent
    ) -> GetModulesEvent:
        """Get modules step."""
        # get input data, store llm into ctx
        task = ev.get("task")
        llm: LLM = ev.get("llm")

        if task is None or llm is None:
            raise ValueError("'task' and 'llm' arguments are required.")

        await ctx.set("llm", llm)

        # format prompt and get result from LLM
        prompt = SELECT_PRMOPT_TEMPLATE.format(
            task=task, reasoning_modules=_REASONING_MODULES
        )
        result = llm.complete(prompt)

        return GetModulesEvent(task=task, modules=str(result))

    @step
    async def refine_modules(
        self, ctx: Context, ev: GetModulesEvent
    ) -> RefineModulesEvent:
        """Refine modules step."""
        task = ev.task
        modules = ev.modules
        llm: LLM = await ctx.get("llm")

        # format prompt and get result
        prompt = ADAPT_PROMPT_TEMPLATE.format(
            task=task, selected_modules=modules
        )
        result = llm.complete(prompt)

        return RefineModulesEvent(task=task, refined_modules=str(result))

    @step
    async def create_reasoning_structure(
        self, ctx: Context, ev: RefineModulesEvent
    ) -> ReasoningStructureEvent:
        """Create reasoning structures step."""
        task = ev.task
        refined_modules = ev.refined_modules
        llm: LLM = await ctx.get("llm")

        # format prompt, get result
        prompt = IMPLEMENT_PROMPT_TEMPLATE.format(
            task=task, adapted_modules=refined_modules
        )
        result = llm.complete(prompt)

        return ReasoningStructureEvent(
            task=task, reasoning_structure=str(result)
        )

    @step
    async def get_final_result(
        self, ctx: Context, ev: ReasoningStructureEvent
    ) -> StopEvent:
        """Gets final result from reasoning structure event."""
        task = ev.task
        reasoning_structure = ev.reasoning_structure
        llm: LLM = await ctx.get("llm")

        # format prompt, get res
        prompt = REASONING_PROMPT_TEMPLATE.format(
            task=task, reasoning_structure=reasoning_structure
        )
        result = llm.complete(prompt)

        return StopEvent(result=result)

from llama_index.core.workflow import ( Workflow, Context, StartEvent, StopEvent, step, ) from llama_index.core.llms import LLM class SelfDiscoverWorkflow(Workflow): """Self discover workflow.""" @step async def get_modules( self, ctx: Context, ev: StartEvent ) -> GetModulesEvent: """Get modules step.""" # get input data, store llm into ctx task = ev.get("task") llm: LLM = ev.get("llm") if task is None or llm is None: raise ValueError("'task' and 'llm' arguments are required.") await ctx.set("llm", llm) # format prompt and get result from LLM prompt = SELECT_PRMOPT_TEMPLATE.format( task=task, reasoning_modules=_REASONING_MODULES ) result = llm.complete(prompt) return GetModulesEvent(task=task, modules=str(result)) @step async def refine_modules( self, ctx: Context, ev: GetModulesEvent ) -> RefineModulesEvent: """Refine modules step.""" task = ev.task modules = ev.modules llm: LLM = await ctx.get("llm") # format prompt and get result prompt = ADAPT_PROMPT_TEMPLATE.format( task=task, selected_modules=modules ) result = llm.complete(prompt) return RefineModulesEvent(task=task, refined_modules=str(result)) @step async def create_reasoning_structure( self, ctx: Context, ev: RefineModulesEvent ) -> ReasoningStructureEvent: """Create reasoning structures step.""" task = ev.task refined_modules = ev.refined_modules llm: LLM = await ctx.get("llm") # format prompt, get result prompt = IMPLEMENT_PROMPT_TEMPLATE.format( task=task, adapted_modules=refined_modules ) result = llm.complete(prompt) return ReasoningStructureEvent( task=task, reasoning_structure=str(result) ) @step async def get_final_result( self, ctx: Context, ev: ReasoningStructureEvent ) -> StopEvent: """Gets final result from reasoning structure event.""" task = ev.task reasoning_structure = ev.reasoning_structure llm: LLM = await ctx.get("llm") # format prompt, get res prompt = REASONING_PROMPT_TEMPLATE.format( task=task, reasoning_structure=reasoning_structure ) result = llm.complete(prompt) return StopEvent(result=result)

运行工作流¶

In [ ]

from llama_index.llms.openai import OpenAI

workflow = SelfDiscoverWorkflow()
llm = OpenAI("gpt-4o")

from llama_index.llms.openai import OpenAI workflow = SelfDiscoverWorkflow() llm = OpenAI("gpt-4o")

In [ ]

from IPython.display import display, Markdown

task = "Michael has 15 oranges. He gives 4 oranges to his brother and trades 3 oranges for 6 apples with his neighbor. Later in the day, he realizes some of his oranges are spoiled, so he discards 2 of them. Then, Michael goes to the market and buys 12 more oranges and 5 more apples. If Michael decides to give 2 apples to his friend, how many oranges and apples does Michael have now?"
result = await workflow.run(task=task, llm=llm)
display(Markdown(str(result)))

from IPython.display import display, Markdown task = "Michael有15个橘子。他给了弟弟4个，并用3个橘子和邻居换了6个苹果。当天晚些时候，他发现一些橘子坏了，于是扔掉了2个。然后，Michael去市场又买了12个橘子和5个苹果。如果Michael决定给朋友2个苹果，那么他现在有多少橘子和苹果？" result = await workflow.run(task=task, llm=llm) display(Markdown(str(result)))

为了解决确定 Michael 现在有多少橘子和苹果的任务，我们将遵循结构化的推理方法，将问题分解成更小、更易于管理的部分。我们将逐步更新橘子和苹果的数量。

逐步计划：¶

1. 最初的橘子数量

   • Michael 最初有 15 个橘子。
   • ( 橙子 = 15 )

2. 给弟弟的橘子

   • Michael 给了弟弟 4 个橘子。
   • ( \text{橘子} = 15 - 4 = 11 )

3. 换成苹果的橘子

   • Michael 用 3 个橘子换了 6 个苹果。
   • ( \text{橘子} = 11 - 3 = 8 )
   • ( \text{苹果} = 0 + 6 = 6 )

4. 扔掉的坏橘子

   • Michael 扔掉了 2 个坏橘子。
   • ( \text{橘子} = 8 - 2 = 6 )

5. 在市场买的橘子

   • Michael 又买了 12 个橘子。
   • ( \text{橘子} = 6 + 12 = 18 )

6. 最初的苹果数量

   • Michael 最初没有苹果。
   • ( \text{苹果} = 6 ) (来自交易)

7. 在市场买的苹果

   • Michael 又买了 5 个苹果。
   • ( \text{苹果} = 6 + 5 = 11 )

8. 给朋友的苹果

   • Michael 给了朋友 2 个苹果。
   • ( \text{苹果} = 11 - 2 = 9 )

最终数量：¶

• 橘子 18
• 苹果 9

因此，Michael 现在有 18 个橘子和 9 个苹果。