Regularization

Regularization techniques for neural networks.

nabla.nn.utils.regularization.l1_regularization(params, weight=0.01)

Compute L1 (Lasso) regularization loss.

L1 regularization adds a penalty equal to the sum of absolute values of parameters. This encourages sparsity in the model parameters.

Parameters:

• params (list[Array]) – List of parameter arrays (typically weights)

• weight (float) – Regularization strength

Returns:

Scalar L1 regularization loss

Return type:

Array

nabla.nn.utils.regularization.l2_regularization(params, weight=0.01)

Compute L2 (Ridge) regularization loss.

L2 regularization adds a penalty equal to the sum of squares of parameters. This encourages small parameter values and helps prevent overfitting.

Parameters:

• params (list[Array]) – List of parameter arrays (typically weights)

• weight (float) – Regularization strength

Returns:

Scalar L2 regularization loss

Return type:

Array

nabla.nn.utils.regularization.elastic_net_regularization(params, l1_weight=0.01, l2_weight=0.01, l1_ratio=0.5)

Compute Elastic Net regularization loss.

Elastic Net combines L1 and L2 regularization: ElasticNet = l1_ratio * L1 + (1 - l1_ratio) * L2

Parameters:

• params (list[Array]) – List of parameter arrays (typically weights)

• l1_weight (float) – L1 regularization strength

• l2_weight (float) – L2 regularization strength

• l1_ratio (float) – Ratio of L1 to L2 regularization (0 = pure L2, 1 = pure L1)

Returns:

Scalar Elastic Net regularization loss

Return type:

Array

nabla.nn.utils.regularization.dropout(x, p=0.5, training=True, seed=None)

Apply dropout regularization.

During training, randomly sets elements to zero with probability p. During inference, scales all elements by (1-p) to maintain expected values.

Parameters:

• x (Array) – Input array

• p (float) – Dropout probability (fraction of elements to set to zero)

• training (bool) – Whether in training mode (apply dropout) or inference mode

• seed (int | None) – Random seed for reproducibility

Returns:

Array with dropout applied

Return type:

Array

nabla.nn.utils.regularization.spectral_normalization(weight, u=None, n_iterations=1)

Apply spectral normalization to weight matrix.

Spectral normalization constrains the spectral norm (largest singular value) of weight matrices to be at most 1. This stabilizes training of GANs.

Parameters:

• weight (Array) – Weight matrix to normalize [out_features, in_features]

• u (Array | None) – Left singular vector estimate (updated during training)

• n_iterations (int) – Number of power iterations to approximate largest singular value

Returns:

Tuple of (normalized_weight, updated_u)

Return type:

tuple[Array, Array]

nabla.nn.utils.regularization.gradient_clipping(gradients, max_norm=1.0, norm_type='l2')

Apply gradient clipping to prevent exploding gradients.

Parameters:

• gradients (list[Array]) – List of gradient arrays

• max_norm (float) – Maximum allowed gradient norm

• norm_type (str) – Type of norm to use (“l2” or “l1”)

Returns:

Tuple of (clipped_gradients, total_norm)

Return type:

tuple[list[Array], Array]