A Multi-Faceted Optimization Algorithm for Deep Learning Alternative to Gradient Descent

Gradient Descent is the popular optimization algorithm used in deep learning and it is stated as a first-order optimization algorithm. The parameters updates are performed infirst order derivatives. The direction ofthe steepest descent is identified by updating the parameters in the opposite direction of the objective function. The major disadvantage is that it will slow the process for the larger dataset and once all the examples processed, then only learning is performed. To overcome the above drawback, as an alternative to gradient descent, the Huddle PSO is proposed. The Huddle PSO is a modified form of PSO, where a single swarm is divided into multiple sub swarms and it is compatible with optimization problems having multiple constraints, to balance both exploration and exploitation. Each sub swarm focuses on a specific region, diversification method is to choose and initiate the sub swarms, the main aim is to increase diversity and concentrate on the optimal value. TSP is solved using PSO by preserving diversity that is by updating the capacity of memory to every particle in PSO and thereby performance is improved for smaller TSP problems.