On solution of the Kostyuchenko problem (Q1760515)

In the Hilbert space \(L^{2}(0,\pi)\), the author studies the basis property of a system of Kostyuchenko type: \[ \mathrm{S}_{\alpha}^{+}\equiv\{e^{i\alpha n t}\sin n t\}_{n\in \mathbb{N}}. \] Note, that the system \( \mathrm{S}_{\alpha}\equiv\{e^{i\alpha n t}\sin n t\}_{n\in \mathbb{Z}} \) is a collection of eigenfunctions of the quadratic pencil \[ -y''(t)+2i\alpha\lambda y'(t)+(1-\alpha^{2})\lambda^{2}y(t)=0, \quad t\in [0,\pi], \quad y(0)=y(\pi)=0. \] It is known that in the Hilbert space \(L^{2}(0,\pi)\) for \(\alpha\in (-1,1)\) the system \(\mathrm{S}_{\alpha}^{+}\) is complete and for \(\alpha\in (-1/\sqrt{2},1/\sqrt{2})\) it forms a Riesz basis [\textit{A. A. Shkalikov}, Math. USSR, Sb. 63, No. 1, 97--119 (1989); translation from Mat. Sb., Nov. Ser. 135(177), No. 1, 96--118 (1988; Zbl 0668.47010)]. In the present paper by function-theoretic methods, the author establishes a stronger result: the system \(\mathrm{S}_{\alpha}^{+}\) forms a Riesz basis in the Hilbert space \(L^{2}(0,\pi)\) if \[ -\frac{e^{\pi\sqrt{3}}-1}{e^{\pi\sqrt{3}}+1}<\alpha<\frac{e^{\pi\sqrt{3}}-1}{e^{\pi\sqrt{3}}+1}, \quad \alpha\neq \pm\frac{e^{\pi}-1}{e^{\pi}+1}. \] To prove this, the author investigates the system \[ \{v_{n}^{\pm}\}_{n\geqslant m}\equiv \{a(t)\varphi_{n}(t)\pm b(t)\psi_{n}(t)\}_{n\geqslant m}, \] where \(a\), \(b\), \(\varphi_{n}\) and \(\psi_{n}\) are complex-valued functions in general.