Siddhant Academy

comprehensive overview of Regularization

Siddhant Bhattarai's photo
Siddhant Bhattarai
·

5 min read

1. What is Regularization?

Regularization is a technique used in machine learning to prevent overfitting by adding a penalty to the model's complexity. It discourages the model from fitting too closely to the training data, making it more generalizable to unseen data.

In simpler terms, regularization adds a constraint or regularizer to the loss function to keep the model weights small.

2. Why is Regularization Important?

  • Overfitting Prevention: Prevents the model from learning noise in the data.

  • Improves Generalization: Helps the model perform well on unseen data.

  • Simpler Models: Encourages the model to prefer simpler solutions with smaller weights.

3. Types of Regularization

A. L1 Regularization (Lasso Regression)

  • Loss Function:

  • Description: Adds a penalty proportional to the absolute value of the coefficients.

  • Effect: Can shrink some coefficients to zero, effectively performing feature selection.

B. L2 Regularization (Ridge Regression)

  • Loss Function:

  • Description: Adds a penalty proportional to the square of the coefficients.

  • Effect: Does not shrink coefficients to zero but forces them to be small, reducing multicollinearity issues.

C. Elastic Net Regularization

  • Loss Function:

  • Description: Combines both L1 and L2 regularization.

  • Effect: Provides a balance between feature selection (L1) and shrinkage (L2).

4. Mathematical Background of Regularization

Regularization modifies the cost function by adding a penalty term:

  • λ (lambda): Regularization parameter that controls the strength of the penalty.

    • λ = 0: No regularization (acts like ordinary least squares regression).

    • λ → ∞: Model is highly constrained (can lead to underfitting).

In L1 regularization, the penalty is based on the L1 norm (sum of absolute values), while in L2 regularization, the penalty is based on the L2 norm (sum of squares).

5. Key Factors to Consider Before Using Regularization

  1. Choice of Regularization Type:

    • L1 is useful for feature selection (zeroes out irrelevant features).

    • L2 is useful for when all features are relevant but need smaller weights.

    • Elastic Net is useful when you want a mix of L1 and L2.

  2. Hyperparameter Tuning:

    • The regularization strength (λ) is a hyperparameter that should be tuned (e.g., using cross-validation).

  3. Feature Scaling:

    • Regularization is sensitive to feature scales. Normalize or standardize your data to improve performance.

  4. Sparse Data:

    • L1 regularization works well with sparse data (data with many zeroes), such as text features in NLP.

6. Types of Problems Regularization Solves

  • Overfitting: Prevents the model from memorizing the noise in the training data.

  • High-dimensional data: Regularization helps handle datasets where the number of features is large compared to the number of observations.

  • Multicollinearity: L2 regularization helps by stabilizing the coefficient estimates.

7. Applications of Regularization

  • Finance: Prevents overfitting in stock price predictions.

  • Healthcare: Improves model performance in predictive health outcomes when dealing with a large number of features.

  • NLP (Natural Language Processing): Handles sparse data in text classification.

  • Computer Vision: Regularization prevents overfitting in CNN (Convolutional Neural Networks).

8. Advantages and Disadvantages of Regularization

Advantages

  • Prevents overfitting: Improves generalization to unseen data.

  • Handles high-dimensional data: Helps when the dataset has many irrelevant features.

  • Feature selection: L1 regularization can perform automatic feature selection.

Disadvantages

  • Tuning required: The regularization parameter (λ) needs to be carefully tuned.

  • Bias introduction: Too much regularization can cause underfitting.

  • Feature scaling required: Regularization may require standardized input data for optimal performance.

9. Performance Metrics for Regularized Models

The same metrics used in linear regression are used here:

1. Mean Absolute Error (MAE)

  • Measures the average magnitude of errors.

2. Mean Squared Error (MSE)

  • Penalizes larger errors more heavily than smaller errors.

3. Root Mean Squared Error (RMSE)

  • Provides errors in the same units as the dependent variable.

4. R-Squared (R^2)

  • Represents the proportion of variance in the target variable explained by the model.

10. Python Code Example for Regularization

Here’s a complete Python example demonstrating Ridge (L2) Regularization and Lasso (L1) Regularization:

Sample Dataset

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge, Lasso
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Sample data (more samples for robust testing)
data = pd.DataFrame({
    'Feature1': [1, 2, 3, 4, 5, 6, 7],
    'Feature2': [2, 4, 6, 8, 10, 12, 14],
    'Target': [5, 7, 9, 11, 13, 15, 17]
})

# Splitting features and target
X = data[['Feature1', 'Feature2']]
y = data['Target']

# Train-test split with larger test size to ensure enough samples in the test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Ridge Regression (L2)
ridge = Ridge(alpha=1.0)  # λ = 1.0
ridge.fit(X_train, y_train)
ridge_pred = ridge.predict(X_test)

# Lasso Regression (L1)
lasso = Lasso(alpha=0.1)  # λ = 0.1
lasso.fit(X_train, y_train)
lasso_pred = lasso.predict(X_test)

# Metrics for Ridge
print("Ridge Regression:")
print(f"MAE: {mean_absolute_error(y_test, ridge_pred)}")
print(f"MSE: {mean_squared_error(y_test, ridge_pred)}")
print(f"R² Score: {r2_score(y_test, ridge_pred)}\n")

# Metrics for Lasso
print("Lasso Regression:")
print(f"MAE: {mean_absolute_error(y_test, lasso_pred)}")
print(f"MSE: {mean_squared_error(y_test, lasso_pred)}")
print(f"R² Score: {r2_score(y_test, lasso_pred)}")

11. Summary

Regularization is a powerful tool to improve model performance by controlling overfitting and handling high-dimensional data. The choice of L1 (Lasso), L2 (Ridge), or Elastic Net depends on the problem at hand:

  • Use L1 (Lasso) for feature selection.

  • Use L2 (Ridge) when you want to stabilize coefficients.

  • Use Elastic Net when you want the benefits of both.

By mastering regularization, you can build robust machine learning models that generalize well to unseen data.