You’ve heard the hype and want the real picture: how do neural networks work? Layers of simple units multiply inputs by weights, add them up, and get tuned on examples until predictions are right.
If you are new to the field, start with what is artificial intelligence for the wider map, then come back here for the mechanics.
| Item | Value |
|---|---|
| Parameters in a network with 64 inputs, 32 hidden neurons and 10 outputs | 2,410 parameters |
| Input neurons needed for an 8×8 image | 64 neurons |
| Learning rate that trained well in one experiment | 0.1 |
| Learning rate that broke training in the same experiment | 3.0 |
| Score gap of an overfitted large network on new data | 10 points lower |
| Year Frank Rosenblatt built the perceptron | 1958 |
These figures come from the Neurohive step-by-step guide, and the experiments are that guide’s own. Results change with different data and settings, so check the source before you rely on them.
What is a neural network?
A neural network is a machine learning model built from simple connected units called neurons. Each neuron takes a few numbers, combines them and passes the result on. Connected in layers, thousands or billions of them can turn pixel values into a word, or a prompt into the next word of an answer, as the Neurohive guide explains.
The idea is old. In 1943 Warren McCulloch and Walter Pitts described a mathematical model of a nerve cell, and in 1958 Frank Rosenblatt built the perceptron, the first neural network that could learn from examples, according to the same guide.
Nobody writes the rules by hand. You show the network examples with correct answers, such as photos labelled dog or cat, and a training algorithm adjusts the connections. That is supervised learning, described by the Neurohive guide as the most common way to train neural networks.
The contrast with learning from unlabelled data is covered in this site’s guide to supervised vs unsupervised learning.
Inside a single neuron
The Neurohive guide says an artificial neuron does three things:
- It multiplies each input by a weight, which shows how important that input is and whether it pushes the output up or down.
- It adds up the products together with a bias.
- It passes that sum through an activation function, and the result becomes an input for the next layer.
Weights and biases are the network’s parameters. They start as small random numbers and are what training changes, and the InDepth guide defines training as the process of finding useful values for them.
The bias shifts the function a neuron computes, so the network can fit data that does not pass through zero. It plays the same role as the intercept in a straight-line fit.
The activation function is what makes depth useful. Without it, any number of layers collapses into a single linear function, as the Neurohive guide puts it.
- ReLU is the most popular choice in hidden layers: f(z) = max(0, z), so negative values become zero.
- Sigmoid returns a probability between 0 and 1, which suits yes-or-no answers, according to Pluralsight.
- Softmax is used in output layers for choosing among several classes.
Layers: from raw input to an answer
Neurons are organised in layers, and the Neurohive guide says every neural network has three kinds of them:
- Input layer: receives the raw data, one neuron per feature, such as 64 neurons for the pixels of an 8×8 image. It computes nothing and just passes values on.
- Hidden layers: everything in between. Each hidden neuron learns some intermediate feature, like an edge or a stroke in an image.
- Output layer: produces the answer, for example ten neurons with softmax for the digits 0-9.
A network with many hidden layers is called a deep neural network, which is where the term deep learning comes from. If the terms blur together, the page on machine learning vs deep learning sorts them out.
In a fully connected layer, every neuron connects to every neuron of the previous layer, so parameters are easy to count: the weights plus one bias per neuron. For a network with 64 inputs, 32 hidden neurons and 10 outputs, the Neurohive guide gives this tally:
- Hidden layer: 64 × 32 + 32 = 2,080 parameters.
- Output layer: 32 × 10 + 10 = 330 parameters.
- Total: 2,410 parameters, while a large language model has billions.
The forward pass: how an input becomes a prediction
In the forward pass, values flow from the input layer through the hidden layers to the output. Each layer computes weighted sums, adds biases, applies the activation and hands the result to the next layer.
The Neurohive guide walks through a tiny network with 2 inputs, 3 hidden neurons with ReLU and 1 output neuron with sigmoid. The input is (0.5, 0.8) and the correct answer is 1.
- The first hidden neuron has weights (0.2, -0.4) and bias 0.1, so its weighted sum is 0.2 × 0.5 – 0.4 × 0.8 + 0.1 = -0.12.
- ReLU turns that negative value into 0, so this neuron stays silent. The other two hidden neurons give 0.43 and 0.15.
- The output neuron combines the three values with its own weights and bias, gets 0.099, and the sigmoid turns it into 0.525.
The network therefore thinks the answer is 1 with a probability of 52.5%. The correct answer is 1, so the prediction is weak, and training should push it higher.
How does a network learn from its mistakes?
Learning repeats three steps, as the Neurohive guide summarises:
- A forward pass makes a prediction.
- A loss function measures the error.
- Backpropagation with gradient descent nudges every weight to reduce that error.
The InDepth guide explains that backpropagation computes the derivatives, called gradients, and an optimizer such as gradient descent uses them to update the parameters.
A common picture of gradient descent is a ball rolling down a hill, as the import learn ai newsletter describes it. The same source notes that the parameters are updated over many iterations called epochs.
The learning rate decides how big each nudge is. In the Neurohive guide’s experiment, 0.1 trained well, 0.001 barely moved and 3.0 broke training completely. Treat that as one demonstration, not a universal setting.
Overfitting and the main network types
Bigger is not automatically better. In the Neurohive experiment, a large network memorized 60 noisy training points perfectly but scored 10 points lower on new data, and weight decay closed the gap. This is overfitting: the network learns the quirks of its training examples instead of the pattern.
Different data also calls for different architectures. As of Pluralsight’s May 29, 2024 guide, the common types and their niches look like this:
- Convolutional networks (CNN): primarily used for image and video recognition.
- Recurrent networks (RNN): best for sequential data, such as time series analysis.
- Generative adversarial networks (GAN): used for generating new data that resembles the training data.
- Transformers: designed to handle sequential data like text, and ChatGPT is built using a transformer, according to the Pluralsight guide.
Frequently asked questions
Do neural networks work like the human brain?
Show answer
Only loosely. The Neurohive guide says modern networks share only the basic idea with the brain: signals flow through many simple units, and the strength of each connection is what the network learns.
Why does a network need an activation function?
Show answer
Without nonlinear activations, each layer computes an affine function, and composing those layers still gives one affine function regardless of depth, according to the InDepth guide. A nonlinear activation lets the network represent more complex relationships.
What is backpropagation in one sentence?
Show answer
It is the step that computes how the loss changes with each parameter, so that an optimizer such as gradient descent can update the weights and biases.
How many parameters can a network have?
Show answer
The small example above has 2,410, while the Neurohive guide notes that a large language model has billions.
