Orbit closed permutation groups, relation groups, and simple groups (Q6156186)

The action of a group \(G\) of permutations on a set \(\Omega\) induces an action of \(G\) on the power set \(P(\Omega)\) of \(\Omega\). The group \(G\) is called orbit closed if any permutation of \(\Omega\) which preserves the orbits of \(G\) on \(P(\Omega)\) lies in \(G\). The group \(G\) is called a relation group if there is a set \(R \subseteq P(\Omega)\) such that \(G\) is the group of all permutations of \(\Omega\) mapping \(R\) to itself. In this paper, the authors conjecture that for all but a ``controllable'' set of permutation groups it holds that if the group is orbit closed, then it is a relation group. The authors verify the conjecture for simple groups and their subgroups. This result strengthens the evidence provided in the earlier paper [the authors, J. Graph Theory 101, No. 1, 29--51 (2022; \url{doi:10.1002/jgt.22808})] where the conjecture was verified for abelian groups or more generally subgroups of direct products of regular permutation groups. Along the way, the authors prove several structural results about simple orbit closed and relation permutation groups. One motivation for the work is taken from research on the symmetry groups of Boolean functions in \(n\) variables with \(G\) permuting the variables. This setting is equivalent to studying symmetry groups of hypergraphs.