Manuscript received April 7, 2024; revised June 18, 2024; accepted June 28, 2024; published November 26, 2024
Abstract—This paper proposes the optimization of the
Transformer model for analysis of sequential data using a
modified clonal selection algorithm (mCSA). Transformers
demonstrate better performance over Long Short-Term
Memory (LSTM) deep networks when the input sequence is
exceptionally long. They are good at capturing long-term
dependencies in comparison to LSTM networks. However, this
comparison is valid only if the hyperparameters are optimized
correctly. Also, transformers are very sensitive to their
hyperparameters. Designing the architecture of the
transformer model for better performance is very complex and
time-consuming. There have been other efforts using Bayesian,
Grid Search, Blackbox, and metaheuristic optimization
techniques for the optimization of the architecture of deep
learning models. mCSA is a nature-inspired immunocomputing
approach. The performance of the optimized transformer
model has been compared with an unoptimized transformer
model, genetic algorithm optimized transformer, Clonal
Selection Algorithm optimized LSTM(CSA_LSTM), Clonal
Selection Optimized Hybrid Convolutional Neural Network and
LSTM network (CSA-CNN-LSTM, and Random Forest search
algorithm. CSA optimized transformer model has consistently
shown better performance in comparison to all other models for
a variety of datasets such as IMDB-movie, SMS-Spam, and US
Twitter Airline datasets. Here we also show that improper
optimization of transformer hyperparameters can lead to
degraded performance that cannot surpass even traditional ML
approaches like random forest. We have also carried out
ablation studies to understand the impact of various
hyperparameters on the performance of our model.
Keywords—deep learning, clonal selection algorithm,
immunocomputing, genetic algorithm, hyperparameters
optimization, nature inspired algorithm, topology optimization,
transformers
Cite: Ashish Kharel and Devinder Kaur, "Optimizing the Topology of Transformer Networks Using Modified Clonal Selection Algorithm: A Bio-Inspired Immunocomputing Approach," International Journal of Machine Learning vol. 14, no. 5, pp. 119-128, 2024.
Copyright © 2024 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).