HMath.Optimization.FirstOrder¶

Description

Explicit optimizer state for deterministic gradients, SGD and minibatches. Defaults: beta1=.9,beta2=.999,epsilon=1e-8; RMSProp/AdaDelta rho=.9; AdamW decays weights by .01*learningRate. Bias corrections use one-based steps.

On this page¶

API	Purpose
GradientMethod	First-order optimizer selector, including momentum and adaptive-moment variants.
GradientDescent	Declaration and type contract below.
Momentum	Declaration and type contract below.
Nesterov	Declaration and type contract below.
AdaGrad	Declaration and type contract below.
RMSProp	Declaration and type contract below.
AdaDelta	Declaration and type contract below.
Adam	Declaration and type contract below.
AdamW	Declaration and type contract below.
AMSGrad	Declaration and type contract below.
AdaMax	Declaration and type contract below.
Nadam	Declaration and type contract below.
RAdam	Declaration and type contract below.
OptimizerState	Opaque accumulated first-order optimizer moments/history.
initializeOptimizer	Allocate zero momentum/moment buffers for a nonnegative parameter count.
optimizerStep	Apply one checked stateful update using the supplied gradient and positive rate.
minimizeGradient	Minimize with explicit gradient/state and fixed learning rate.
minimizeGradientWithCallback	Observe each finite iterate after its objective/gradient evaluation.
stochasticGradient	One pass over explicit sample/batch gradients; ordering and shuffling are caller-owned, allowing exact replay.
miniBatchGradient	Average a nonempty explicit set of same-length checked sample gradients at the current parameters.
projectedGradient	Project every update and stop on the projected-gradient mapping, including constrained boundary optima.
proximalGradient	Apply prox(rate,point) after each gradient step; residual is the proximal-gradient mapping norm.
fista	Accelerated proximal gradient using the Beck-Teboulle t recurrence and a fresh mapping residual at accepted points.
coordinateDescent	Cyclic coordinates or blocks; every coordinate must occur exactly once.

Documentation

GradientMethod

data GradientMethod Source #

First-order optimizer selector, including momentum and adaptive-moment variants. Stateful parameters are validated by the selected optimizer.

Constructors

GradientDescent
Momentum
Nesterov
AdaGrad
RMSProp
AdaDelta
Adam
AdamW
AMSGrad
AdaMax
Nadam
RAdam

Instances

Instances details

Bounded GradientMethod Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods minBound :: GradientMethod # maxBound :: GradientMethod #
Enum GradientMethod Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods succ :: GradientMethod -> GradientMethod # pred :: GradientMethod -> GradientMethod # toEnum :: Int -> GradientMethod # fromEnum :: GradientMethod -> Int # enumFrom :: GradientMethod -> [GradientMethod] # enumFromThen :: GradientMethod -> GradientMethod -> [GradientMethod] # enumFromTo :: GradientMethod -> GradientMethod -> [GradientMethod] # enumFromThenTo :: GradientMethod -> GradientMethod -> GradientMethod -> [GradientMethod] #
Show GradientMethod Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods showsPrec :: Int -> GradientMethod -> ShowS # show :: GradientMethod -> String # showList :: [GradientMethod] -> ShowS #
Eq GradientMethod Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods (==) :: GradientMethod -> GradientMethod -> Bool # (/=) :: GradientMethod -> GradientMethod -> Bool #

OptimizerState

data OptimizerState Source #

Opaque accumulated first-order optimizer moments/history. Initialize and advance through the stateful optimizer API.

Instances

Instances details

Show OptimizerState Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods showsPrec :: Int -> OptimizerState -> ShowS # show :: OptimizerState -> String # showList :: [OptimizerState] -> ShowS #
Eq OptimizerState Source #
Instance details Defined in HMath.Optimization.FirstOrder Methods (==) :: OptimizerState -> OptimizerState -> Bool # (/=) :: OptimizerState -> OptimizerState -> Bool #

initializeOptimizer

initializeOptimizer :: Int -> Either MathError OptimizerState Source #

Allocate zero momentum/moment buffers for a nonnegative parameter count.

optimizerStep

optimizerStep :: GradientMethod -> Double -> OptimizerState -> [Double] -> [Double] -> Either MathError ([Double], OptimizerState) Source #

Apply one checked stateful update using the supplied gradient and positive rate. Nesterov here applies a corrected momentum direction; minimizeGradient evaluates the actual look-ahead gradient.

minimizeGradient

minimizeGradient :: GradientMethod -> OptimizeOptions -> Objective -> Gradient -> [Double] -> Either MathError OptimizeResult Source #

Minimize with explicit gradient/state and fixed learning rate. Only finite gradient residuals below tolerance certify convergence.

minimizeGradientWithCallback

minimizeGradientWithCallback :: (OptimizeResult -> Either MathError Bool) -> GradientMethod -> OptimizeOptions -> Objective -> Gradient -> [Double] -> Either MathError OptimizeResult Source #

Observe each finite iterate after its objective/gradient evaluation. False stops explicitly with convergence=False; callback failures propagate. Callback observations do not themselves count as objective or derivative evaluations.

stochasticGradient

stochasticGradient :: GradientMethod -> Double -> [Gradient] -> [Double] -> Either MathError ([Double], OptimizerState) Source #

One pass over explicit sample/batch gradients; ordering and shuffling are caller-owned, allowing exact replay. All gradients are checked before update.

miniBatchGradient

miniBatchGradient :: [Gradient] -> Gradient Source #

Average a nonempty explicit set of same-length checked sample gradients at the current parameters.

projectedGradient

projectedGradient :: OptimizeOptions -> Objective -> Gradient -> ([Double] -> Either MathError [Double]) -> [Double] -> Either MathError OptimizeResult Source #

Project every update and stop on the projected-gradient mapping, including constrained boundary optima.

proximalGradient

proximalGradient :: OptimizeOptions -> Objective -> Gradient -> (Double -> [Double] -> Either MathError [Double]) -> [Double] -> Either MathError OptimizeResult Source #

Apply prox(rate,point) after each gradient step; residual is the proximal-gradient mapping norm.

fista

fista :: OptimizeOptions -> Objective -> Gradient -> (Double -> [Double] -> Either MathError [Double]) -> [Double] -> Either MathError OptimizeResult Source #

Accelerated proximal gradient using the Beck-Teboulle t recurrence and a fresh mapping residual at accepted points.

coordinateDescent

coordinateDescent :: [[Int]] -> OptimizeOptions -> Objective -> Gradient -> [Double] -> Either MathError OptimizeResult Source #

Cyclic coordinates or blocks; every coordinate must occur exactly once.