What is Loss Function in Machine Learning

Loss Function in Regression Models

In this section, we discuss the loss functions used in regression models.

MSE (Means Square Error)

MSE is one of the most famous and commonly used loss functions in regression analysis. For this loss function, the mean square of the difference between the predicted and actual values is calculated:

MSE uses the squared values to create a loss functions. Therefore, the loss function is parabolic for prediction (or error) values. Its advantages are being easy to understand and having only one local minimum. its disadvantages include not being robust to outliers.

Mean Absolute Error (MAE)

Mean Absolute Error, or MAE, is another loss function with interesting properties. This loss function, like MSE, uses the difference between the predicted and actual value as a criterion but does not take into account the direction of the difference. Therefore, MAE calculates the mean absolute value of the differences between the predicted and actual values. MAE advantages include its intuitiveness, ease of use, and robustness to outliers. Its main disadvantage is that we can not directly use gradient descent, and have to rely on sub-gradient calculations.

Huber Loss

Huber loss function is less affected by outliers than MSE. Also, unlike MAE loss function, minimization is easily achievable. Since different values of δ in the Huber loss equation change the shape of the loss function, choosing the right value for it is a sensitive and difficult task. If δ is too large, this loss function will be the same as the MSE loss function. If δ approaches zero, the Huber loss function approaches the MAE loss function.