Improvements on the kernel estimation in line transect sampling without the shoulder condi\-tion (Q5952091)

\textit{Y. P. Mack} et al. [Commun. Stat., Theory Methods 28, No. 10, 2277--2296 (1999; Zbl 1072.62663)] proposed to use the boundary kernel method to estimate the wildlife population density from the line transect data when the detection function does not satisfy the shoulder condition, the first derivative of the detection function at distance zero is 0. It was demonstrated in their paper that the boundary kernel estimator (with the use of the optimal end point kernel developed by \textit{S. Zhang} and \textit{R. J. Karunamuni} [J. Stat. Plann. Inference 70, No. 2, 301--316 (1998; Zbl 0938.62037)] performed significantly better than the reflection method [\textit{E. F. Schuster}, Commun. Stat., Theory Methods 14, 1123--1136 (1985; Zbl 0585.62070)]. However, the boundary kernel method has two main drawbacks: the estimates may be negative and the boundary kernel estimator is always associated with a large variability. These two drawbacks of the boundary kernel method have been noticed by \textit{Mack} et al. [ibid. 28, No. 10, 2277--2296 (1999)], but no remedies were discussed in their paper. \textit{S. Zhang} et al. [J. Am. Stat. Assoc. 94, No. 448, 1231--1241 (1999; Zbl 0994.62029)] offered a new method to correct these two drawbacks of the boundary kernel method. It is shown by Zhang et al. that their proposed method performs better than the boundary kernel method in the sense that it has much smaller mean squared error value than that of the boundary kernel estimator. In this paper, we discuss how to apply their method (it will be called the new method throughout this paper) to the estimation problem in line transect sampling. It is observed that the new method is superior to the boundary kernel method from both asymptotic theoretical results and simulation results.