6 Mins Read

What is a Neural network?

What is a Neural network

What is a Neural network?

A neural network is a supervised learning algorithm originally developed by Frank Rosenblatt in 1957.  Neural networks are one of the oldest and most primitive models of machine learning. However, the concepts developed in their design are widely applicable to complex deep neural networks.

An overview of a neural network
Figure1. An overview of a neural network.

Artificial and Biological Neurons

There are billions of neurons in the human brain. Neurons are interconnected nerve cells that play a role in transmitting and processing chemical and electrical signals in the human brain. Dendrites are branches that receive information from other nerve cells. The nucleus of the cell processes the information received from the dendrites. Axon is a cable used by neurons to send information. A synapse is a connection between an axon and other nerve dendrites.

Artificial neurons, however, are mathematical functions that simulate biological neurons, where inputs are weighed separately, summed up, and outputs are calculated through non-linear functions. Below is a figure showing artificial and biological neurons.

Schematic of an artificial and biological neuron
Figure2. Schematic of an artificial and biological neuron.

A neuron is an elementary unit in an artificial neural network. it is a mathematical function designed based on the function of biological neurons; weights are assigned to a neuron from one or more separate inputs. The inputs are collected and passed through a nonlinear function (the activation function or ” ϕ”) to produce the output. In fact, an artificial neuron is controlled by the following function, in which Y is the output of the artificial neuron:

The output of artificial neurons by the activation function
Figure3. The output of artificial neurons by the activation function ϕ, with b as a bias.

Neural network architecture

A neural network consists of several artificial neurons. This section goes over the different layers of an artificial neuron. Generally, neural networks are composed of three layers: the input layer, the hidden layer, and the output layer. The hidden layer may be made up of more than one layer.

Input layer: No calculations are performed in this layer, it only passes information to the next layer.

Hidden layer: In the hidden layer, intermediate processing is performed. After performing calculations, the weights (signals or information) are transferred from the input layer to the next layer (another hidden layer or output layer). It’s also possible for a  neural network to not have a hidden layer.

Output layer: This layer uses an activation function to produce the desired output format. (For example, for classification problems, we use the softmax function).

Types of neural networks

Neural networks come in various flavors. It is also possible to divide main categories into subcategories. Here is a list of the most commonly used neural networks.

Feedforward neural network: A feed-forward neural network is an artificial neural network with no loops between the units. There is only one direction in which the information moves in this network: forward. That is, from input nodes through hidden nodes, if any, and to output groups. This type of network does not contain loops or cycles. Feedforward neural networks can be divided into two types:

  • Single layer perceptron: It is the simplest type of neural network without hidden layers. This network is made up of only one layer. In other words, it is the output layer. When counting the layers of a network, we do not include the input layer since no calculations are performed in the input layer. Several weights connect the inputs directly to the outputs; Thus, this network is called a single-layer perceptron.
  • Multi-layer Perceptron (MLP): A feed-forward network consists of several layers of computing units connected to one another in a feed-forward pattern. Neurons in one layer are directly connected to neurons in the next. Multilayer perceptrons are very useful because they can learn non-linear descriptions (in many cases, the data we deal with is not linearly separable).

Convolutional neural network: A convolutional neural network is similar to a regular neural network. They also consist of neurons with trainable weights and biases. In convolutional neural networks (CNNs or ConvNets), the connection pattern of units is inspired by the visual cortex. An area of space called the receptive field allows units to respond to stimuli. Each receptive field partially overlaps the other and covers the entire field of vision. The convolution operator can mathematically approximate the response of each unit. In image and video recognition, recommender systems are the main applications of these networks. A large amount of data is required to train convolutional neural networks.

Recurrent neural networks: In Recurrent Neural Networks (RNN), the connections between units form a directed loop (these networks propagate data forward and backward, from the final processing levels to the initial levels). A temporary dynamic behavior can be seen in this structure. In contrast to feed-forward neural networks, recurrent neural networks can process arbitrary input sequences. As a result, these networks can perform tasks such as handwriting recognition, voice recognition, and time series estimation.

Related articles
Ensemble machine learning is a powerful technique that leverages the strengths of multiple weak learning models, also known as...
Neural Network
Activation functions are the main components of neural network nodes. This article examines the various types of activation functions...
Regularization is a technique used in Machine Learning and Deep Learning models to prevent overfitting. This paper introduces L1,...
Machine Learning Engineers Should Use Docker
Docker is a platform that enables developers to easily create, deploy, and run applications in containers, and has gained...
In this paper, we compare the performance of CPU, GPU, and TPU processors to see which one is better...
History of AI
In the second part of a series of articles about the history of artificial intelligence, we look at important...
Subscribe to our newsletter and get the latest practical content.

You can enter your email address and subscribe to our newsletter and get the latest practical content. You can enter your email address and subscribe to our newsletter.