Comparison of Convex and Non-Convex least square Penalties under Different Criteria for Low and High Dimensional Data

Abstract

Penalized least squares methods are commonly used for simultaneous coefficient estimation and variable selection in high-dimensional linear models. All of these methods have their attractive and efficient variable selection properties. We compare some of penalized least square methods i.e Smoothly Clipped Absolute Deviation (SCAD) and Minimax Concave Penalty (MCP) (non-convex penalties) with Least Absolute shrinkage and Selection Operator (LASSO) and OLS Post-LASSO (convex penalties). The property of an oracle estimate and consistent selection of important variables are fulfilled by smoothly clipped absolute deviation and minimax concave penalty method, whereas least absolute shrinkage and selection operator does not possess oracle property but consistently select the important variable. The efficiency of these techniques are based on the appropriate selection of tuning parameter. Though, Cross Validation (CV), Generalized Cross Validation (GCV), Akaike Information Criteria (AIC) and Bayesian Information Criteria (BIC) are the most commonly used techniques for the selection of the tuning parameter. However, these techniques do not select the tuning parameter properly when they do not meet with their suitable conditions like when the correlation and noise level (standard error) varies or number of predictor diverges than number of observation, then these techniques results in over fitting phenomena or badly perform in variable selection and coefficient estimation. We compare some of convex and non-convex penalties using 10-fold cross validation to select appropriate tuning parameter under different noise levels and a varying number of variables to identify the true variables in the model. We examine the performance of these methods by monte-carlo simulation.