Podcast(Generated by NotebookLM)

My research journey began with an exploration of Partial Differential Equations (PDEs), specifically focusing on traveling wave solutions. Intrigued by the application of the Klein-Gordon equation with hyperbolic functions, I questioned the conventional use of a static beta parameter within these functions. This led me to hypothesize that a dynamically updated beta parameter, mirroring the time-dependent nature of the weight and loss function, could potentially enhance model performance.

To achieve this dynamic update, I sought a suitable function and discovered the Allen-Cahn equation through the Gemini Model. This equation, closely resembling those utilized in reaction-diffusion systems and phase-field models, offered a promising mechanism. Its ability to facilitate the propagation of “phase boundaries” through parameter space suggested potential for inducing sharp transitions within network behavior, an area requiring further investigation. Further exploration of PDE solutions for traveling waves led me to Ginzburg-Landau theory. Recognizing the potential of its free energy calculation as a component within the loss function, I pursued a simplified implementation. This resulted in a novel loss calculation framework directly inspired by a simplified Ginzburg-Landau equation.

Given the physics-based foundation of the neural network components explored thus far, I investigated the feasibility of employing a similarly grounded optimization approach. This led to the identification of Stochastic Gradient Langevin Dynamics (SGLD) as a suitable candidate. The implementation of SGLD, aided by the Claude language model, provided valuable insights into its application and divergences from traditional optimization techniques.

Preliminary results indicate comparable performance to a standard fully connected network. Although the proposed model did not consistently outperform a modified, fully connected network, this work, developed as an exploratory weekend project, provides a foundation for future research into physics-inspired neural network architectures and optimization algorithms.

Introduction: The Intersection of Physics and Machine Learning

Machine learning and physics might seem like two distinct fields, but they share a common goal: understanding and modeling complex systems. In machine learning, we build models to learn from data and make predictions. In physics, we develop theories to explain natural phenomena. By bringing these disciplines together, we can create more powerful and flexible neural networks.

The Inspiration: Ginzburg-Landau Theory

At the heart of experimental neural network lies the Ginzburg-Landau theory, a powerful framework originally developed to describe superconductivity and other phase transitions in physical systems. But what does superconductivity have to do with neural networks, you might ask?

The key insight is this: both superconducting systems and neural networks can be thought of as complex systems trying to find optimal configurations. In superconductors, it’s about finding the state of lowest energy. In neural networks, it’s about finding the configuration that best fits the data.

The Custom Loss Function: A Physics-Inspired Approach

Inspired by the Ginzburg-Landau free energy functional, we’ve designed a custom loss function for this network:

Where:

• \[\phi_{\text{norm}} = \frac{\phi}{\max(|\phi|)}\]
• \[\text{target}_{\text{norm}} = \frac{\text{target\_one\_hot}}{\max(|\text{target\_one\_hot}|)}\]
• \[\frac{\partial \phi_{\text{norm}}}{\partial x} \approx \frac{\phi_{\text{norm}}(i+1) - \phi_{\text{norm}}(i-1)}{2}\]
• \(\alpha\) is the coefficient for the quadratic term that encourages the network’s output to match the target values.
• \(\gamma\) is the coefficient for the quartic term that allows for multiple stable states.
• The spatial term promotes coherence in the network’s outputs across spatial dimensions.

Breakdown of the Loss Function

• Quadratic Term: Encourages the network’s output to match the target values. This is similar to the traditional mean squared error used in many neural networks.
• Quartic Term: Allows for multiple stable states, potentially enabling the network to capture more complex patterns. This term, with a negative sign, adds richness to the energy landscape, facilitating better exploration during training.
• Spatial Term: Promotes coherence in the network’s outputs across spatial dimensions. This is particularly useful for tasks involving spatial data, such as image processing.

Dynamic Weight Modification: Adapting to Input

Another unique feature of this network is its dynamic weight modification scheme:

where:

• \(\mathbf{W}_0\) is the initial weight matrix.
• \(\beta\) is a learned parameter.
• \(t_{\text{dynamic}}\) is a dynamic term that adapts based on the input.

Explanation

This might look complex, but the idea is simple yet powerful: the network adapts its weights based on the input it receives. The use of hyperbolic functions (cosh and sinh) introduces a form of non-Euclidean geometry, potentially allowing the network to better capture hierarchical structures in the data.

Physics-Inspired Optimization: Langevin Dynamics

To train this network, I’ve developed a custom optimizer inspired by Langevin dynamics, a concept from statistical physics used to describe the motion of particles in a fluid:

Step Method Equation

For each parameter \(\theta_i\):

1. Compute the loss gradient with respect to each parameter: \(F_i = -\frac{\partial \mathcal{L}}{\partial \theta_i}\) where \(\mathcal{L}\) is the loss function, and \(F_i\) is the force (negative gradient) applied to \(\theta_i\).

2. Update the velocity: \(v_i \leftarrow (1 - \gamma) v_i + \alpha F_i + \sqrt{2 \gamma \alpha T} \cdot \eta_i\) where \(\eta_i\) is a noise term sampled from a standard normal distribution \(\mathcal{N}(0, 1)\).

3. Update the parameter: \(\theta_i \leftarrow \theta_i + v_i\)

Full Update Equation

For each parameter \(\theta_i\) :

Where:

• \(\alpha\) is the learning rate.
• \(\gamma\) is the damping coefficient.
• \(T\) is the temperature.
• \(\eta_i \sim \mathcal{N}(0, 1)\) is the noise term.

Combined in a Single Expression

Combining both steps, we get the following update rules for each parameter \(\theta_i\):

This optimizer introduces concepts of velocity, damping, and temperature into the learning process. The idea is to allow the network to explore the loss landscape more thoroughly, potentially escaping local minima and finding better global solutions.

Beta Update Equations

Given:

• \(\beta\) is the learnable parameter.
• \(\phi\) is the model output.
• \(\text{target}\) is the ground truth.
• \(dt\) is the time step.
• \(\kappa\) is a constant.

The steps involved in updating \(\beta\) are as follows:

1. Compute the loss derivative with respect to the input: \(\text{d\_loss} = \mathcal{L}(\phi, \text{target})\) where \(\mathcal{L}\) is the loss function.

2. Compute the spatial derivative of the loss: \(\text{d\_loss}_x = \frac{\text{d\_loss}_{i+1} - \text{d\_loss}_{i-1}}{2}\)

3. Compute the force term: \(\text{force} = -\kappa \cdot \text{d\_loss}_x\)

4. Compute the second spatial derivative of \(\beta\): \(\beta_{xx} = \beta_{i+1} - 2\beta_i + \beta_{i-1}\)

5. Compute the time derivative of \(\beta\): \(\beta_t = \frac{\beta - \beta_{\text{previous}}}{dt}\)

6. Update \(\beta\) using the combined terms: \(\beta \leftarrow \beta + dt \left( \beta_t - \beta_{xx} + \beta - \beta^3 + \text{force} \right)\)

Putting these steps together, the beta update rule can be written as:

Putting It All Together: A New Paradigm for Neural Networks

By combining these physics-inspired components - the Ginzburg-Landau-based loss function, dynamic weight modification, and Langevin dynamics-inspired optimization - i’ve created a neural network that operates on principles quite different from traditional architectures.

Potential Benefits

1. Better Handling of Complex, Hierarchical Data Structures: The dynamic weight modification and rich energy landscape help capture intricate patterns in the data.
2. Improved Exploration of the Solution Space: The Langevin dynamics-inspired optimizer enhances the network’s ability to find global optima.
3. Multiple Stable States: The quartic term allows for capturing diverse data patterns.
4. Enhanced Spatial Coherence: Useful for tasks involving spatial data, ensuring smooth transitions in the output.

Challenges

1. Increased Complexity in Training: More parameters and terms to tune can complicate the training process.
2. Need for Careful Interpretation: Understanding the network’s behavior requires a deeper theoretical insight.
3. Potential Instabilities: Non-standard loss functions and optimization processes might introduce new challenges.

Experimental Results

Example Outputs from 1 Layer Experimental NN

1 Layer Experimental Net Example Test Set Output

To provide a clear comparison, here’s how this experimental neural network performs on MNIST dataset across various metrics:

1 Layer Experimental NN with Mish activation and batch norm

2 Layer Experimental NN with Mish activation and batch norm

Visualization with Some Datasets

1 Layer Example on Circle dataset

2 Layer Example on Circle dataset

2 Layer Example on Circle dataset

1 Layer Example on Classification dataset

1 Layer Example on Classification dataset

1 Layer Example on Moons dataset

2 Layer Example on Moons dataset

Code Snippet for the ExperimentalLayer

Here’s a more detailed code snippet of the ExperimentalLayer to give readers a clearer picture of its implementation:

Landau Layer with Traveling Wave Approach

class LandauLayer(nn.Module):
    def __init__(self, input_size, output_size, beta_init, alpha=0.0035, gamma=5.0):
        super(LandauLayer, self).__init__()
        self.w = nn.Parameter(torch.randn(output_size, input_size))
        nn.init.xavier_normal_(self.w)
        self.beta = nn.Parameter(torch.tensor(beta_init).float())
        self.alpha = alpha
        self.gamma = gamma
        self.t = nn.Parameter(torch.tensor([1.0]))

    def forward(self, x):
        t_dynamic = self.t * torch.sigmoid(torch.mean(x, dim=-1)).unsqueeze(-1)
        w2 = self.w * (torch.arange(x.size(-1), dtype=torch.float, device=x.device) * torch.cosh(5 * self.beta) + t_dynamic * torch.sinh(5 * self.beta)).unsqueeze(1)
        phi = torch.matmul(w2, x.unsqueeze(-1)).squeeze(-1)
        return phi
        
    def update_beta(self, phi, target, dt, kappa=0.01):
        d_loss = self.d_loss(phi, target).detach()
        d_loss_x = (torch.roll(d_loss, -1) - torch.roll(d_loss, 1)) / 2.0
        force = -kappa * d_loss_x
        
        with torch.no_grad():
            beta_xx = (torch.roll(self.beta, -1) - 2 * self.beta + torch.roll(self.beta, 1))
            beta_t = (self.beta - self.beta.clone().detach()) / dt
            self.beta.data += dt * (beta_t - beta_xx + self.beta - self.beta**3 + force)
            
    def d_loss(self, phi, target):
        max_abs_phi = torch.max(torch.abs(phi), dim=1, keepdim=True)[0]
        phi_norm = phi / (max_abs_phi)  
        
        phi_x = (torch.roll(phi_norm, -1, dims=0) - torch.roll(phi_norm, 1, dims=0)) / 2.0
        target_one_hot = F.one_hot(target, num_classes=phi.size(1)).float()
        
        max_abs_target = torch.max(torch.abs(target_one_hot), dim=1, keepdim=True)[0]
        target_norm = target_one_hot / (max_abs_target )  
        
        diff = phi_norm - target_norm
        
        quadratic_term = self.alpha * (diff)**2
        quartic_term = self.gamma * (diff)**4
        spatial_term = 0.5 * phi_x**2
        
        d_loss = quadratic_term - quartic_term + spatial_term
        
        mean_d_loss = d_loss.mean()
        return torch.abs(mean_d_loss)

Modified Langevin Optimizer

class LangevinLandauOptimizer:
    def __init__(self, model, learning_rate=0.001, damping=0.1, temperature=0.8):
        self.model = model
        self.lr = learning_rate
        self.damping = damping
        self.temperature = temperature
        self.velocities = [torch.zeros_like(p.data) for p in model.parameters()]

    def step(self, phi, target):
        d_loss = self.model.llayer2.d_loss(phi, target)
        
        for i, (param, velocity) in enumerate(zip(self.model.parameters(), self.velocities)):
            force = -torch.autograd.grad(d_loss, param, create_graph=True)[0]
            
            # Langevin dynamics directly application without modify
            velocity.mul_(1 - self.damping).add_(force * self.lr)
            noise_scale = torch.sqrt(torch.tensor(2 * self.damping * self.temperature * self.lr))
            velocity.add_(torch.randn_like(velocity) * noise_scale)
            
            param.data.add_(velocity)

        return d_loss.item()

Simple Network

class ExperimentalNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, beta_init):
        super(ExperimentalNet, self).__init__()
        self.llayer1 = LandauLayer(input_size, hidden_size, beta_init)
        self.llayer2 = LandauLayer(hidden_size, output_size, beta_init)
        self.bn1 = nn.BatchNorm1d(hidden_size)

    def forward(self, x):
        # x = F.rrelu(self.bn1(self.layer1(x)))
        x = F.rrelu(self.bn1 (self.llayer1(x)))
        x = F.rrelu(self.llayer2(x))
        return x

In the past weeks, an intriguing article was published on text classification. This article aims to provide simple but effective text classification using k-nearest-neighbors (kNN) and gzip, in contrast to Deep Neural Networks (DNNs). Although DNNs yield high results in text classification, they require extensive processing, millions of parameters, and a large number of labeled data. However, this article suggests a lightweight method that delivers good results without requiring any parameters or training. In this post, we will test this application using a dataset consisting of approximately 300 Turkish questions and answers related to law.

Related Studies and Approach

The article also mentions previous studies in this field, which were conducted using minimum cross entropy but did not produce efficient results. The explanations, formulas, and diagrams in this section are taken directly from the article.

Text classification with compressors can be divided into two main approaches:

Using Shannon Information Theory for entropy estimation and approximating Kolmogorov complexity and using information distance. The approach primarily uses a text compression technique called Prediction by Partial Matching (PPM) and applies it for topic classification. This approach estimates the cross-entropy between the probability distribution created with a given document d for a specific class c: Hc(d).

The article’s approach to this topic consists of a lossless compressor, a compressor-based distance metric, and a k-nearest neighbor classifier. Lossless compressors try to represent information using fewer bits, assigning shorter codes to symbols with higher probability.

For example, x1 belongs to the same category as x2, but belongs to a different category than x3. Assuming C(·) represents the compressed length, we will find C(x1x2) — C(x1) < C(x1x3) — C(x1); here, C(x1x2) denotes the compressed length of the combination of x1 and x2.

To measure the shared information content between two objects, Bennett et al. (1998) define the information distance E(x,y) as follows:

E(x,y) equates the similarity with the shortest program length needed to transform one object into the other.

The uncomputability of Kolmogorov complexity hinders the computability of E(x,y), hence Li et al. (2004) propose the Normalized Compression Distance (NCD), using the compressed length C(x) of real-world compressors to approximate Kolmogorov complexity K(x). The formula is as follows:

Here, C(x) represents the length after x has been compressed at the highest rate by the compressor. Generally, the higher the compression rate, the closer C(x) is to K(x).

We can use the kNN with the distance matrix provided by the NCD for classification. Our method can be implemented with 12 lines of Python code:

for (x1, _) in test_set:  
    Cx1 = len(gzip.compress(x1.encode()))  
    distance_from_x1 = []  
    for (x2, _) in train_set:  
        Cx2 = len(gzip.compress(x2.encode()))  
        x1x2 = " ".join([x1, x2])  
        Cx1x2 = len(gzip.compress(x1x2.encode()))  
        ncd = (Cx1x2 - min(Cx1, Cx2)) / max(Cx1, Cx2)  
        distance_from_x1.append(ncd)  
    sorted_idx = np.argsort(np.array(distance_from_x1))  
    top_k_class = [train_set[idx][1] for idx in sorted_idx[:k]]  
    predict_class = max(set(top_k_class), key=top_k_class.count)

Data Preparation and Testing the Approach

This part of the post is a bit experimental, and I believe, rather amateurish. Firstly, I downloaded my ChatGPT data and then split them into questions and answers. I assigned my data as questions, and the responses from GPT as answers, and converted them into a csv file. I then split it into train and test sets, but my approach here was as if I was training a DNN. I then conducted a test, and the results were somewhat successful.

import gzip  
import numpy as np  
import csv  
  
  
dataset = []  
with open('dataset.csv', 'r', newline='', encoding='utf-8') as file:  
    csv_reader = csv.reader(file)  
    next(csv_reader)    
    for row in csv_reader:  
        question = row[0]  
        answer = row[1]    
        dataset.append((question, answer))  
  
train_size = int(0.8 * len(dataset))  
training_set = dataset[:train_size]  
test_set = dataset[train_size:]  
  
def normalized_compression_distance(x1, x2):  
    #C(x) and C(y) are the compressed sizes of strings x and y, respectively.  
    Cx1 = len(gzip.compress(x1.encode()))   
    Cx2 = len(gzip.compress(x2.encode()))  
    x1x2 = " ".join([x1, x2])  
    #C(xy) is the compressed size of the concatenated strings x and y.  
    Cx1x2 = len(gzip.compress(x1x2.encode()))  
    #min{C(x), C(y)} represents the smallest compressed size between x and y.  
    #max{C(x), C(y)} represents the largest compressed size between x and y.  
    ncd = (Cx1x2 - min(Cx1, Cx2)) / max(Cx1, Cx2)  
    return ncd  
  
# Function to predict the class for a given text using KNN  
def predict_class_knn(text, dataset, k=3):  
    distances = []  
    for (question, answer) in dataset:  
        distance = normalized_compression_distance(text, question)  
        distances.append((distance, answer))  
    distances.sort(key=lambda x: x[0])  
    top_k_class = [distances[i][1] for i in range(k)]  
    predicted_class = max(set(top_k_class), key=top_k_class.count)  
    return predicted_class

I tried random things since I didn’t know and couldn’t remember exactly what my conversations were about, but I couldn’t get a fully efficient result. However, I was surprised at one point because when I left a blank or entered meaningless letters or numbers, I received this response

Afterward, I collected approximately 300 data points from a law website and decided to give it a try, and the result was surprising. I felt as if I was performing a sort of search, but the result was good. It gave a lot of wrong results in some places, but for now, it was satisfactory.

I also tested model on Sentiment Analysis with winvoker’s Turkish Sentiment Analysis Dataset and results are very good. It works quickly and effectively.

import gzip  
import numpy as np  
import datasets  
  
  
dataset = datasets.load_dataset("winvoker/turkish-sentiment-analysis-dataset", split="train")  
  
  
questions = dataset['text']  
answers = dataset['label']  
  
  
data_tuples = list(zip(questions, answers))  
  
train_size = int(0.8 * len(data_tuples))  
training_set = data_tuples[:train_size]  
test_set = data_tuples[train_size:]  
  
      
def normalized_compression_distance(x1, x2):  
    #C(x) and C(y) are the compressed sizes of strings x and y, respectively.  
    Cx1 = len(gzip.compress(x1.encode()))   
    Cx2 = len(gzip.compress(x2.encode()))  
    x1x2 = " ".join([x1, x2])  
        #C(xy) is the compressed size of the concatenated strings x and y.  
    Cx1x2 = len(gzip.compress(x1x2.encode()))  
    #min{C(x), C(y)} represents the smallest compressed size between x and y.  
    #max{C(x), C(y)} represents the largest compressed size between x and y.  
    ncd = (Cx1x2 - min(Cx1, Cx2)) / max(Cx1, Cx2)  
    return ncd  
  
    # Function to predict the class for a given text using KNN  
def predict_class_knn(text, dataset, k=3):  
    distances = []  
    for (question, answer) in dataset:  
        distance = normalized_compression_distance(text, question)  
        distances.append((distance, answer))  
    distances.sort(key=lambda x: x[0])  
    top_k_class = [distances[i][1] for i in range(k)]  
    predicted_class = max(set(top_k_class), key=top_k_class.count)  
    return predicted_class

Article

“Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors (Jiang et al., Findings 2023)

https://github.com/bazingagin/npc_gzip/tree/main

In the context of neural networks, this article gives a thorough analysis and use of the Gravitational Search Algorithm (GSA). The GSA is an optimization method that borrows principles from nature and imitates the effects of mass and gravity. To maximize the weights of the network, the technique is combined with a feed-forward neural network. A Python implementation of the GSA for optimizing the weights of a neural network with a single hidden layer is shown in the provided code. Using matplotlib, the evaluation of the algorithm’s performance is done over a number of epochs. Scholarly studies exploring the use of GSA in neural networks and its benefits over conventional optimization techniques are among the sources considered for this study.

INTRODUCTION

The Gravitational Search Algorithm (GSA) is a nature-inspired optimization algorithm that mimics the law of gravity and mass interactions. In 2009, Rashedi, Nezamabadi-pour, and Saryazdi first presented it. The GSA has been applied to a variety of domains, including machine learning, to improve the weights of neural networks. In order to optimize the weights of a neural network with a single hidden layer, the GSA is presented in this article with a Python implementation.

PROBLEM DEFINITION

The primary challenge in training neural networks is the optimization of the weights. Traditional optimization methods, such as gradient descent, can often get stuck in local minima, leading to suboptimal performance. The GSA, being a global optimization algorithm, can potentially overcome this issue by exploring the solution space more thoroughly. The problem at hand is to implement the GSA in Python and use it to optimize the weights of a neural network.

LITERATURE ANALYSIS

Several studies have been conducted on the application of the GSA in neural networks. For instance, a study by Fedorovici et al. (2012) presented a method of embedding GSA in CNNs for OCR systems. The study demonstrated that the GSA, in combination with the Back Propagation BP algorithm, could improve performance by avoiding the algorithms’ traps in local minima. Another study by George and Huerta (2018) introduced Deep Filtering, a method for end-to-end time-series signal processing, based on a system of two deep convolutional neural networks, designed for classification and regression to rapidly detect and estimate parameters of signals in highly noisy time-series data streams. The study showcased the application of this method for the detection and parameter estimation of gravitational waves from binary black hole mergers.

METHODS AND TECHNOLOGIES TO BE USED

The method used in this study is the Gravitational Search Algorithm, a nature-inspired optimization algorithm. The GSA is implemented in Python and used to optimize the weights of a neural network with a single hidden layer. The neural network is a simple feed-forward network implemented using the numpy library. Since the GSA is a global optimization technique, it might locate the global minimum of the error function rather than a local minimum. Given that the error function of a neural network is frequently non-convex and may have numerous local minima, this makes it an excellent option for optimizing the weights of a neural network. The literature has looked into the application of GSA for neural network weight optimization. For instance, a way of embedding GSA in CNNs for OCR systems was provided in a study by Fedorovici et al. (2012). The study showed that by avoiding the algorithms’ traps in local minima, the GSA and the Back Propagation method might increase performance. Deep Filtering is a technique for end-to-end time-series signal processing developed by George and Huerta (2018). It is based on a system of two deep convolutional neural networks and was created for classification and regression in order to quickly identify and estimate the parameters of signals in time-series data streams with high levels of noise. This method’s use for the detection and parameter determination of gravitational waves from binary black hole mergers was demonstrated in the study.

STUDIES

The neural network is a simple feed-forward network with a single hidden layer. The weights of the network are initialized randomly, and the GSA is used to optimize these weights over a series of epochs. The performance of the algorithm is evaluated by calculating the error between the network’s output and the desired output.

The parameters for the GSA and the neural network are set. This includes the number of agents , the learning rate, the number of iterations , the number of input features, and the number of neurons in the hidden layer. The agents, which represent the weights of the neural network, are initialized randomly. Each agent is a vector of weights, and the size of this vector is determined by the number of input features and the number of neurons in the hidden layer. The input data and the desired output are set. In this case, the input data is a vector of 0.2s, and the goal prediction is 0.9. A function is defined to calculate the fitness of an agent.

The fitness is calculated as the sum of the squared differences between the output of the network (using the agent’s weights) and the goal prediction.

 def calculate_fitness(agent):  
    # Reshape the agent's weights to match the network's structure  
    weights = agent.reshape((n_inputs, n_hidden))  
    biases = np.zeros(n_hidden)  # You could also optimize the biases  
  
    # Forward pass through the network  
    hidden_layer = np.maximum(0, np.dot(input_data, weights) + biases)  
    output_layer = np.dot(hidden_layer, output_weights)  
  
    # Calculate error  
    error = np.sum((output_layer - goal_pred) ** 2)  
    return error

A function is defined to calculate the gravitational forces between agents. The force between two agents is proportional to the product of their masses and inversely proportional to the distance between them. The direction of the force is from the agent with lesser mass to the agent with greater mass.

 def calculate_forces(i):  
    forces_i = np.zeros(agents.shape[1])  
    for j in range(n_agents):  
        if i != j:  
            # Distance between agents  
            distance = np.abs(agents[i] - agents[j])  
  
            # Gravitational force  
            force = (mass[i] * mass[j]) / (distance + 1e-5)  # Add a small constant to avoid division by zero  
  
            # Direction  
            direction = (agents[j] - agents[i]) / (distance + 1e-5)  # Add a small constant to avoid division by zero  
  
            # Update force  
            forces_i += force * direction  
    return forces_i

The algorithm starts by initializing a population of agents. Each agent represents a potential solution to the optimization problem. In the context of neural networks, each agent is a vector of weights. The fitness of each agent is evaluated. The fitness function depends on the specific problem being solved. In the context of neural networks, the fitness function could be the error between the network’s output (using the agent’s weights) and the desired output. 
The mass of each agent is calculated based on its fitness. Agents with better fitness have higher mass. The mass of an agent i at iteration t is calculated as follows:

The gravitational force between two agents is calculated. The force between agent i and agent j at iteration t is calculated as follows:

The positions of the agents are updated based on the forces. The new position of agent i at iteration t+1 is calculated as follows:

The weights of the neural network, which are represented by the agents in the GSA, are updated in each epoch based on the calculated forces and a random velocity. The formula used in the code to update the weights is as follows:

In this example parameters are in the below

# GSA parameters  
n_agents = 100  # Number of agents (solutions)  
# lr = 0.001  # Learning rate  
lr = np.float32(0.001)   
  
epochs = 1000  # Number of iterations  
n_inputs = 20  # The number of input features  
n_hidden = 10  # The number of neurons in the hidden layer  
  
# Initialize agents (weights)  
# agents = np.random.uniform(low=-1, high=1, size=(n_agents, n_inputs * n_hidden))  
agents = np.random.uniform(low=-1, high=1, size=(n_agents, n_inputs * n_hidden)).astype(np.float32)  
  
# Training data  
# goal_pred = 0.9  
goal_pred = np.float32(0.9)  
# input_data = np.full(n_inputs, 0.2)  
input_data = np.full(n_inputs, 0.2, dtype=np.float32)

REFERENCES

1. Fedorovici, L. O., Precup, R. E., Dragan, F., David, R. C., & Purcaru, C. (2012). Embedding Gravitational SearchAlgorithms in Convolutional Neural Networks for OCR applications. 2012 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI). (https://ieeexplore.ieee.org/document/6249989)
2. 1. George, D. J., & Huerta, E. A. (2018). Deep neural networks to enable real-time multimessenger astrophysics. Physical Review D, 97(4)(https://doi.org/10.1103/physrevd.97.044039)
3. Poma, Y., Melin, P., González, C. M., & Martínez, G. (2019). Optimization of Convolutional Neural Networks Using the Fuzzy Gravitational Search Algorithm. Journal of Applied Mathematics and Robotics. (https://doi.org/10.14313/jamris/1-2020/12)
4. Wang, J., & Han, S. (2015). Feed-Forward Neural Network Soft-Sensor Modeling of Flotation Process Based on Particle Swarm Optimization and Gravitational Search Algorithm. Mathematical Problems in Engineering, 2015. (https://doi.org/10.1155/2015/147843)
5. Nagra, A. A., Alyas, T., Abdul Hamid, M. A., Tabassum, N., & Ahmad, A. (2022). Training a Feedforward Neural Network Using Hybrid Gravitational Search Algorithm with Dynamic Multiswarm Particle Swarm Optimization. Complexity, 2022.(https://doi.org/10.1155/2022/2636515)
6. Ezzat, D., Hassanien, A. E., & Ella, H. A. (2021). An optimized deep learning architecture for the diagnosis of COVID-19 disease based on gravitational search optimization. Applied Soft Computing, 98. (https://doi.org/10.1016/j.asoc.2020.106742)