Caustics for inner and outer billiards (Q1901805)

Let \(T\) be a closed convex curve in the Euclidean plane. This paper considers two dynamical systems associated to \(T\): the inner, (classical) billiard and the outer billiard maps. With natural choices of coordinates in the phase spaces, both billiards become area preserving twist maps of the annulus. Area preserving maps of the annulus have been studied extensively and it is most interesting to interpret the general properties of these maps in the setting of billiards. For example, the invariant circles for the twist maps are the caustics for the billiard maps (respectively, the inner and the outer caustics; the use of the term ``outer caustic'' seems to appear here for the first time). The structure of caustics is an important subject of investigation. In particular, it is interesting to know under which conditions there exist convex caustics. There are existence results due to V. F. Lazutkin and J. Mather. In this paper, the authors study existence of convex caustics and their possible location, in terms of the shape of the curve \(T\). In particular, they prove that if the curvature of \(T\) is not bounded away either from 0 or infinity, then there are no convex inner caustics. Under suitable curvature assumptions, they establish the existence of some open regions which are free from inner caustics, and they estimate from below the area of these regions (again, in terms of the curvature of \(T)\). For outer billiards, the main results concern the Birkhoff instability regions.