Orbit equivalence and permutation groups defined by unordered relations. (Q424390)

Let \(\Omega\) be a finite set. A subset \(R\) of the power set \(P(\Omega)\) is called an unordered relation, and the group preserving \(R\) is defined to be \(\mathcal G(R):=\{g\in\mathrm{Sym}(\Omega)\mid x^g\in R\text{ for all }x\in R\}\) [see \textit{D. Betten}, Mitt. Math. Ges. Hamb. 10, 317-324 (1977; Zbl 0425.20030)]. A group of the form \(G=\mathcal G(R)\) for some unordered relation \(R\) is called a relation group. This paper is an extension of the report [the authors, in Ischia group theory 2008. Proceedings of the conference in group theory, Naples, Italy 2008. Hackensack: World Scientific. 56-67 (2009; Zbl 1208.20001)] characterizing relation groups. A set \(x\in P(\Omega)\) is called a regular set for \(G\leq\mathrm{Sym}(\Omega)\) if the setwise stabilizer of \(x\) equals \(1\), and \(G\) is called orbit closed if the only permutations which leave the orbits of \(G\) set-wise fixed are those which lie in \(G\). The following Basic Lemma is proved: suppose \(H\leq\mathrm{Sym}(\Omega)\) has a regular set \(w\) and either (i) \(H=\mathcal G(R)\) for some \(R\) such that no subset in \(R\) has size \(|w|\), or (ii) \(H\) is not set-transitive but every subgroup of \(\mathrm{Sym}(\Omega)\) properly containing \(H\) is set-transitive; then each subgroup of \(H\) is a relation group and is orbit closed. Using this the authors show that a primitive group \(G\leq\mathrm{Sym}(\Omega)\) is a relation group if and only if it is orbit closed, and that the only cases where \(G\) is not a relation group are when \(G=\mathrm{Alt}(\Omega)\) or \(|\Omega|\leq 10\) (the exceptional cases of small degree are listed). The situation for imprimitive groups is less clear cut, but there is a sense in which most imprimitive groups are relation groups, and the authors give various criteria for this to happen. The classification [Bull. Lond. Math. Soc. 29, No. 6, 697-704 (1997; Zbl 0892.20002)] by \textit{Á. Seress} of primitive groups which contain no regular set plays a crucial role in the proofs.