The accuracy of least squares calculations with the Cholesky algorithm (Q911243)

An extensive study is made of the accuracy with which the Cholesky (or square root) method computes the solution to the normal equation of least squares. The influence of scaling and centering in the polynomial regression model on accuracy of estimation is investigated. It is demonstrated using computer evaluations that scaling does not improve the accuracy. It is shown that loss of accuracy with the Cholesky method is traceable directly to cancellation of common leading digits in subtraction. The double Cholesky algorithm with appropriate centering is proposed and investigated too. Simple formulas for prediction accuracy from the maximum variance inflation factor are evaluated, methods which are improved accuracy are discussed.