Neuroscience

Our natural networks of neurons work as decision-making systems with information processing units called neurons. For example, when we touch a hot plate, neurons are responsible for carrying the information regarding how hot or cold the plate is, deciding if our hands should be moved and then taking the necessary information related to that decision to our muscles and tendons. Artificial neural networks (ANNs) somehow work in a similar fashion, although they have many differences from our natural neural networks. Dissimilar to having a giant neural network system in our bodies that take care of all decision-making, active or reactive, ANNs are designed to be problem-specific. For example, we have ANNs for image classification, drug-target interaction prediction, credit risk estimation, object detection, etc. Artificial neural networks can have different types of information processing layers, such as convolutional and fully connected layers. Fully connected layers of neural networks (Figure 1) are designed to handle structured data like characteristics of houses in house price prediction (such as a number of rooms and bathrooms), features related to clients in credit risk estimation (like salary and number of households), fingerprints of molecules in drug-target interaction prediction, etc. ANNs for supervised learning has one input, one output and one or multiple hidden layers. The input layer is nothing other than the features of data points used for modeling. The number of neurons in the output layer is also determined depending on the problem at hand. For example, in the case of binary classification, two neurons are considered in the output layer. The number and size of hidden layers are variable and can be optimized to result in better performance of ANNs. In this post, we focus on the utility of ANNs in binary classification. However, ANNs can be used for regression, dimensionality reduction, generative modeling, reinforcement learning and more. Figure 1. Schematic representation of a three-layer neural network and an individual neuron as the building block of the network. Neurons as building blocks of neural networks Neurons in artificial neural networks play as information collectors, transformers and propagators in ANNs (Figure 1). Each neuron receives a weighted sum of output values from other neurons, usually in the previous layer, applies linear or nonlinear transformation on the received sum of values and then outputs the resulting value to other neurons, usually in the next layer. The weights used in the input value calculation of neurons are the learned weights (parameters) of ANNs in the training process. The nonlinear transformations are applied through predetermined activation functions. ANNs are known for coming up with complicated nonlinear relationships between input feature values and outputs. Although complexity is not the goal for machine learning modeling, it makes ANNs flexible in figuring out (maybe) all kinds of relationships between inputs and outputs. In fully connected ANNs, activation functions applied on information received in neurons are responsible for that complexity or flexibility. Examples of activation functions are shown in Figure 2. Figure 2. Schematic representation and mathematical formulation of sigmoid, ReLU, Leaky ReLU and ELU activation functions. Forward versus backward propagation Supervised learning has two main processes: 1) Predicting outputs and 2) Learning from the predictions. In ANNs, predictions happen in forward propagation. Weights of ANN between input and first hidden layer are used to calculate input values of neurons of first hidden layer and similarly for other layers in the ANN (Figure 3). Going from input to output is called forward propagation or forward pass, which generates the output values (predictions) for each data point. Then in the backward propagation or backward pass, ANN uses the predicted outputs and their differences with actual outputs to adjust its weights, resulting in better predictions. Figure 3. Forward and backward propagation in neural network modeling. Here we briefly talked about ANN in supervised learning and focused on its use in binary classification. In our next post of the series, we will talk about optimization in neural network modelling. In this series, we plan to introduce several other fundamental topics associated with Machine Learning, such as graph neural networks and transfer learning! Stay tuned !! Editor: Andreas Windemuth & Chinmaya Sadangi

Neuroscience

Deep learning as intelligent communication between neurons