Human Reflection

Summary: The agent collects feedback from humans to refine the plan, to effectively align with the human preference.

Context: Agents create plans and strategies that decompose users’ goals and requirements into a pool of tasks. The tasks will be completed by other tools and agents.

Problem: How to ensure human preference is fully and correctly captured and integrated into the reasoning process and generated plans?

Forces:

  • Alignment to human preference. Agents are expected to achieve users’ goals ultimately, consequently, it is critical for agents to comprehend users’ preferences.
  • Contestability. If the agent’s outputs do not satisfy users’ requirements and will cause negative impacts, there should be a timely process for users to contest the responses of agent.

Solution: Fig. 1 presents a high-level graphical representation of human-reflection. When a user prompts his/her goals and specified constraints, the agent first creates a plan consisting of a series of intermediate steps. The constructed plan and its reasoning process can be presented to the user for review, or sent to other human experts to validate the feasibility and usefulness. The user or expert can provide comments or suggestions to indicate which steps can be updated or replaced. The plan will be iteratively assessed and improved until it is approved by the user/expert.

Figure 1. Plan reflection pattern.

Benefits:

  • Alignment to human preference. The agent can directly receive feedback from users or additional human experts to understand human preferences, and improve the outcomes or procedural fairness, diversity in the results, etc.
  • Contestability. Users or human experts can challenge the agent’s outcomes immediately if abnormal behaviours or responses are found.
  • Effectiveness. Human-reflection allows agents to include users’ perspectives for plan refinement, which can help formalise responses tailored to users’ specific needs and level of understanding. This can ensure the usability of strategies, and improve the effectiveness for achieving users’ goals.

Drawbacks:

  • Fairness preservation. The agent may be affected by users who provide skewed information about the real world.
  • Limited capability. Agents may still have limited capability to understand human emotions and experiences.
  • Underspecification. Users may provide limited or ambiguous reflective feedback to agents.
  • Overhead. Users may need to pay for the multiple rounds of communication with the agent.

Known uses:

  • Inner Monologue [1]. Inner Monologue is implemented in a robotic system, which can decompose users’ instructions into actionable steps, and leverage human feedback for object recognition.
  • Ma et al. [2] explore the deliberation between users and agents for decision-making. Users and agents both need to provide related evidence and arguments for their conflicting opinions.
  • Wang et al. [3] incorporate human feedback for agents to capture the dynamic evolution of user interests and consequently provide more accurate recommendations.

Related patterns:

  • Prompt/response optimiser. Human-reflection can provide human preference and suggestions to improve the generated prompts and responses.
  • Multi-path plan generator. Multi-plan generator creates plans with various directions, and human-reflection can help finalise the plan with user feedback to determine the choice of each intermediate step.
  • Incremental model querying. Human-reflection is enabled by incremental model querying for iterative communication between users/experts and the agent.

References:

[1] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608, 2022.

[2] S. Ma, Q. Chen, X. Wang, C. Zheng, Z. Peng, M. Yin, and X. Ma, “Towards human-ai deliberation: Design and evaluation of llm-empowered deliberative ai for ai-assisted decision-making,” arXiv preprint arXiv:2403.16812, 2024.

[3] Y. Wang, Z. Liu, J. Zhang, W. Yao, S. Heinecke, and P. S. Yu, “Drdt: Dynamic reflection with divergent thinking for llm-based sequential recommendation,” arXiv preprint arXiv:2312.11336, 2023.