Rectifier (neural networks)

Plot of the rectifier (blue) and softplus (green) functions near x = 0

In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument:

$f(x)=x^{+}=\max(0,x)$,

where x is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering. This activation function was first introduced to a dynamical network by Hahnloser et al. in 2000 with strong biological motivations and mathematical justifications. It has been demonstrated for the first time in 2011 to enable better training of deeper networks, compared to the widely-used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical counterpart, the hyperbolic tangent. The rectifier is, as of 2018, the most popular activation function for deep neural networks.

A unit employing the rectifier is also called a rectified linear unit (ReLU).

Rectified linear units find applications in computer vision and speech recognition using deep neural nets.

Softplus

A smooth approximation to the rectifier is the analytic function

$f(x)=\log(1+e^{x}),$

which is called the softplus or SmoothReLU function. The derivative of softplus is $f'(x)={\frac {e^{x}}{1+e^{x}}}={\frac {1}{1+e^{-x}}}$, the logistic function. The logistic function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

$LSE_{0}^{+}(x_{1},...,x_{n}):=LSE(0,x_{1},...,x_{n})=\log \left(1+e^{x_{1}}+\cdots +e^{x_{n}}\right).$

The LogSumExp function itself is:

$\mathrm {LSE} (x_{1},\dots ,x_{n})=\log \left(e^{x_{1}}+\cdots +e^{x_{n}}\right),$

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

Variants

Noisy ReLUs

Rectified linear units can be extended to include Gaussian noise, making them noisy ReLUs, giving

$f(x)=\max(0,x+Y)$, with $Y\sim {\mathcal {N}}(0,\sigma (x))$

Noisy ReLUs have been used with some success in restricted Boltzmann machines for computer vision tasks.

Leaky ReLUs

Leaky ReLUs allow a small, positive gradient when the unit is not active.

$f(x)={\begin{cases}x&{\mbox{if }}x>0\\0.01x&{\mbox{otherwise}}\end{cases}}$

Parametric ReLUs

Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters.

$f(x)={\begin{cases}x&{\mbox{if }}x>0\\ax&{\mbox{otherwise}}\end{cases}}$

Note that for $a\leq 1$, this is equivalent to

$f(x)=\max(x,ax)$

and thus has a relation to "maxout" networks.

ELUs

Exponential linear units try to make the mean activations closer to zero which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.

$f(x)={\begin{cases}x&{\mbox{if }}x>0\\a(e^{x}-1)&{\mbox{otherwise}}\end{cases}}$

$a$ is a hyper-parameter to be tuned and $a\geq 0$ is a constraint.

Advantages

• Biological plausibility: One-sided, compared to the antisymmetry of tanh.
• Sparse activation: For example, in a randomly initialized network, only about 50% of hidden units are activated (having a non-zero output).
• Better gradient propagation: Fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions.
• Efficient computation: Only comparison, addition and multiplication.
• Scale-invariant: $\max(0,ax)=a\max(0,x){\mbox{ for }}a\geq 0$.

Rectifying activation functions were used to separate specific excitation and unspecific inhibition in the Neural Abstraction Pyramid, which was trained in a supervised way to learn several computer vision tasks. In 2011, the use of the rectifier as a non-linearity has been shown to enable training deep supervised neural networks without requiring unsupervised pre-training. Rectified linear units, compared to sigmoid function or similar activation functions, allow for faster and effective training of deep neural architectures on large and complex datasets.

Potential problems

• Non-differentiable at zero; however it is differentiable anywhere else, and a value of 0 or 1 can be chosen arbitrarily to fill the point where the input is 0.
• Non-zero centered
• Unbounded
• Dying ReLU problem: ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies." This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high. It may be mitigated by using Leaky ReLUs instead, which assign a small positive slope to the left of x = 0.

See also

• Softmax function
• Sigmoid function
• Tobit model