Lynoxo Tech

This is based on the research paper by OpenAI here, and we shall use this research to hit two birds with one stone – explain WHAT neural networks are, and HOW they work.

What Are Neural Networks?

Neural networks are a type of machine learning model inspired by the human brain. Just as our brains consist of interconnected neurons that communicate with each other, neural networks are composed of artificial neurons, known as nodes, which are linked together in layers. These networks can learn patterns from data, enabling them to perform tasks like image recognition, natural language processing, and even playing games.

How Do Neural Networks Work?

A neural network operates in three main layers: the input layer, hidden layers, and the output layer.

1. Input Layer: This is where the network receives data. Each node in this layer represents a feature of the input data. For example, in an image recognition task, the input layer might consist of pixel values.
2. Hidden Layers: These layers process the input data through a series of mathematical operations. The hidden layers are where the real magic happens, as the network learns to identify patterns and relationships within the data. The more hidden layers, the deeper the network, hence the term “deep learning.”
3. Output Layer: The final layer produces the output, which could be a classification, a prediction, or some other result depending on the task.

Formatting Data for Neural Networks

Neural networks require data to be formatted in a way that they can process effectively. Typically, this data is structured in tables or matrices. Here’s a basic overview:

1. Structured Table Data: Imagine you have a table of data with rows and columns. Each row represents a single data instance (e.g., a sample image or a data point), and each column represents a feature or attribute of that instance. For example, if you’re using a neural network to predict house prices, each row might contain features like square footage, number of bedrooms, and location, with the price being the target output.
2. Input Data: In a neural network, the input data consists of features that the network will use to make predictions. Each feature corresponds to a column in your structured data. For instance, if you have a table with columns for Size, Bedrooms, and Location, these columns are your inputs. Each row in the table represents an individual data point with values for these features.
3. Output Data: The output data (also known as the target or label) is what the network is trying to predict. In supervised learning, this data is included in your dataset alongside the inputs. For house price prediction, the output would be the price itself.

Weights and Biases in Neural Networks

Neural networks learn from data through adjustments made to weights and biases. Here’s how they work:

1. Weights: Each connection between neurons in different layers has a weight associated with it. Weights determine the strength and direction of the influence one neuron has on another. When you input data into the network, each feature is multiplied by its corresponding weight. The weighted inputs are then summed to contribute to the neuron’s activation. Weights are initially set randomly but are adjusted during training to minimize prediction errors.
2. Biases: Biases are additional parameters added to the weighted sum of inputs. They allow the activation function to shift and adapt, helping the network learn more complex patterns. Each neuron has a bias term, which is added to the weighted sum before applying the activation function.

How Weights and Biases Are Decided

1. Initialization: At the start of training, weights and biases are initialized with small random values. This randomness helps in breaking symmetry and ensures that neurons learn different features.
2. Training: During the training process, the neural network uses algorithms like gradient descent to adjust weights and biases. The goal is to minimize the error between the predicted output and the actual target values. This is done by calculating the gradient (rate of change) of the error with respect to each weight and bias and then updating them in the direction that reduces the error.
3. Optimization: Optimizers are algorithms that help in adjusting weights and biases efficiently. Common optimizers include Stochastic Gradient Descent (SGD), Adam, and RMSprop. They use techniques like momentum and adaptive learning rates to improve the learning process.

When working with neural networks, we first break down our total data into smaller chunks, like pieces of a puzzle. Each chunk represents a set of features (like characteristics of a house) and the outcome we’re trying to predict (like the house price). We feed these chunks into the network one at a time. As the network processes each chunk, it adjusts its internal settings, called weights and biases, to improve its predictions. This training process is repeated with all chunks, gradually refining the network to make better predictions as it learns from each piece of data.

The formula y=ϕ(w⋅x+b) is usually used to describe this entire process where the neuron takes inputs, adjusts them with weights and bias, applies a nonlinearity (ϕ), and then produces an output

Why Nonlinearity is Important

The parameters w, x and b are simple enough to understand as follows:

• w – It is a “weight” parameter, and denotes the importance of the input x. It is initialized by a random number, and is subsequently optimised using gradient descent
• x – The input. This would be, for example, a number (if the data had only one feature) or a vector (say, the height, weight and age of a person)
• b – The bias is a constant value added to the result of the weighted sum. It’s like an extra push to help the neuron make better decisions
  • Small Random Values: Biases are often initialized with small random values close to zero. This ensures that the network doesn’t start with biases that are too large, which could lead to instability in learning.
  • Zero Initialization: In some cases, biases might be initialized to zero. This can be acceptable because biases are adjusted during training and don’t suffer from the same symmetry issues as weights.

However, you may be wondering why we need the nonlinearity function ϕ and why it is needed.

Linear vs. Nonlinear: If we only used linear operations (like weighted sums) in every layer of a neural network, the entire network would just be a complicated linear model, regardless of how many layers it has. This means it could only capture linear relationships in the data, which limits its ability to solve more complex problems.

An often used non linear function is ReLU (Rectified Linear Unit).

ReLU’s Role

ReLU adds nonlinearity by applying the rule: if the input is positive, keep it; if it’s negative, set it to zero. This allows the network to model more complex functions and relationships.

Zeroing Out Negatives: ReLU does turn all negative inputs into zero, which might seem like it’s flattening or losing information. However, this behavior is actually beneficial:

• Sparsity: By zeroing out negative values, ReLU makes the network’s activations sparse. Sparse activations mean that fewer neurons are firing, which can make the network more efficient and help prevent overfitting.
• Efficient Learning: ReLU allows the network to learn more efficiently by focusing on the most important patterns and ignoring less useful information (which often results in negative values).

In essence, ReLU doesn’t flatten the inputs in a harmful way. Instead, it filters out less useful information, helping the network focus on the patterns that matter most, which improves its ability to learn complex relationships.

Weight Normalization Technique

Training Neural Networks – When training a neural network, the goal is to adjust the weights (w) and biases (b) of each neuron so that the network’s predictions get closer to the desired outputs. This is done using an optimization method called stochastic gradient descent (SGD), which gradually adjusts the weights and biases to minimize the error (loss function) between the network’s predictions and the actual results.

Instead of directly adjusting the weight vector ( w ) during training, the authors propose a different approach. They suggest reparameterizing ( w ) in terms of two new parameters: a parameter vector ( v ) and a scalar ( g ). This means that the weight vector ( w ) is now expressed as:

w = (g/|v|)v

Here’s what each part means:

( v ): This is a new vector that has the same number of dimensions as ( w ).

( g ): This is a scalar value, meaning it’s just a single number.

( |v| ): This represents the Euclidean norm (or length) of the vector ( v ), which is basically the distance from the origin to the point represented by ( v ) in space.

Effect of Reparameterization: This reparameterization fixes the length (or norm) of the weight vector ( w ) to be equal to ( g ), regardless of the values of ( v ). In other words, the length of ( w ) doesn’t change during training; only its direction does. This is why it’s called weight normalization.

Why Do This?

• Stabilizing Training: By keeping the length of ( w ) fixed, weight normalization helps stabilize the training process. It prevents the weights from becoming too large or too small, which can lead to more stable and faster convergence during training.
• Improved Convergence: This reparameterization can make the optimization process more efficient, potentially leading to faster convergence, meaning the network learns to make accurate predictions more quickly.

In summary, the authors propose using weight normalization as a way to make the training of neural networks more stable and efficient by reparametrizing the weights in terms of a direction vector ( v ) and a scalar ( g ), keeping the length of the weights fixed during the training process.

For information on how ChatGPT works, CLICK HERE.