diffGrad

diffGrad implements an adaptive gradient-based optimizer that adjusts per-parameter step sizes using differences between consecutive gradients to improve training convergence of deep neural networks, particularly convolutional neural networks (CNNs).

Key Features:

• Adaptive Step Sizes: Dynamically adjusts learning rates for each parameter, assigning larger steps to parameters with rapidly changing gradients and smaller steps to parameters with stable gradients.
• Gradient Difference Utilization: Uses the difference between the current gradient and the immediate past gradient to modulate per-parameter updates.
• Convergence Analysis: Provides a theoretical convergence analysis based on a regret bound within the online learning framework.
• Empirical Performance: Demonstrates superior convergence properties on synthetic complex nonconvex functions and in image categorization experiments.
• Benchmark Comparisons: Evaluated against SGDM, AdaGrad, AdaDelta, RMSProp, AMSGrad, and Adam in comparative experiments.
• Architecture and Activation Robustness: Validated on ResNet-based CNN architectures and shown to maintain performance across different activation functions.

Scientific Applications:

• Image Classification: Training and optimization of ResNet-based CNNs for image categorization on CIFAR10 and CIFAR100 datasets.
• Optimizer Benchmarking: Evaluation on synthetic complex nonconvex functions to assess and compare convergence behavior of optimization algorithms.
• Activation Function Studies: Assessing optimizer robustness across different neural network activation functions.

Methodology:

Per-parameter updates are modulated using the difference between the current gradient and the immediate past gradient to adapt step sizes, and convergence is analyzed via a regret bound within the online learning framework.