Preference-based opponent shaping in differentiable games (Q6868231)

This paper explores multi-agent reinforcement learning (MARL) in general-sum differentiable games, where agents' interdependent strategies can lead to suboptimal equilibria due to unmodeled opponent behaviors.\N\NTraditional MARL methods often assume static opponent preferences, limiting their effectiveness in mixed cooperative-competitive settings like prisoner's dilemma or Stackelberg games. The authors introduce preference-based opponent shaping (PBOS), a method that incorporates preference parameters into agents' loss functions, allowing each agent to weight its own and opponents' losses. This leads to adaptive behavior, promoting cooperation when beneficial, while updating preferences alongside strategies. Experiments in various differentiable games, show convergence to superior equilibria with better reward distributions than baselines.\N\NIn real-world multi-agent systems where equilibria depend on evolving preferences, PBOS offers a scalable way to enhance learning efficiency and fairness without assuming rationality. It bridges gaps in MARL for general-sum games, promoting cooperative strategies that avoid lose-lose traps, emphasizing behavioral adaptability.