Activations

relu(x)

Apply the ReLU activation function.

sigmoid(x)

Apply a piecewise linear approximation to x.

sigmoid_bool(x)

Apply the piecewise softmax function with 2 splits for binary classification.

softmax(x::Array)

Apply the softmax operation. As we do not require the gradients, this is simply checking if each input is_argmax, if true, return the index label.