Assessing the Quality of Ordinary Least Squares in General Lp Spaces
In the context of regression analysis, standard estimation techniques are dominated by the Ordinary Least Squares (OLS) method which yields unbiased, consistent, and efficient estimators when the classical assumptions are satisfied. However, the presence of outliers can significantly drag estimators away from their true parameters even when modeling with the OLS method. The OLS method is implicitly defined on L2 spaces, which implies that large residuals have a disproportionally large, i.e., squared, influence on the regression estimators.
We propose a generalized regression function capable of producing estimators that are closer to the true parameters than OLS’s estimators when the residuals are non-normally distributed and when outliers are present. We achieve this improvement by minimizing the norm of the errors in general Lp spaces, as opposed to minimizing the norm of the errors in the L2 space. The generalized model proposed here—the Ordinary Least Powers model (OLP)—can implicitly adjust its sensitivity to outliers by changing its parameter p, the exponent of the absolute value of the residuals. Especially for residuals of large magnitudes, such as those stemming from outliers, different values of p will implicitly exert different relative weights on the corresponding residual observation.
We fitted OLS and OLP models on simulated data under varying distributions providing outlying observations and compared the mean squared errors (MSE) relative to the true parameters. We examined the differences in the MSEs between the OLS and OLP methods to determine which estimators were more accurate under different conditions. We found that OLP models with smaller p's produce estimators closer to the true parameters when the probability distribution of the error terms is exponential or Cauchy, and larger p's produce closer estimators to the true parameters when the error terms are distributed uniformly. In turn, when the true errors are normally distributed, the best estimators are obtained by using p’s closer or equal to 2, which coincides mathematically with the OLS method. Overall, our results accord well with intuition. The key takeaway is that the OLP method proposed here can produce better estimators (in terms of lower MSE) than the standard OLS method when that residuals are distributed non-normally or in the presence of outliers.
While few studies have explored Lp spaces when dealing with outliers and missing data in specific applications of regression analysis, no other study, to the best of our knowledge, has systematically analyzed, via simulation, the probabilistic distributional conditions, and assumptions necessary for improvement over the standard OLS regression methods.