HMath.DeepLearning.Layers¶

Description

Row-major CPU layers. Convolution/pooling layout is N,C,spatial (1–3 spatial axes), weights are Cout,Cingroups,kernel, or Cin,Coutgroups,kernel for transpose convolution. Padding is symmetric and zero-filled. Attention layout is sequence,feature; True in a mask means allowed. Fully masked rows produce exact zero outputs and weights. No ambient training state exists.

On this page¶

API	Purpose
Training	Layer mode: Training updates/uses batch statistics; Inference uses retained running statistics.
Training	Declaration and type contract below.
Inference	Declaration and type contract below.
dropout	Inverted dropout.
spatialDropout	Sample one inverted dropout mask per N,C pair and share it across every spatial position.
BatchNormState	Per-channel running mean/variance and incorporated training-batch count for batch normalization.
BatchNormState	Declaration and type contract below.
runningMean	Per-channel running mean used during inference.
runningVariance	Per-channel running variance used during inference.
batchesSeen	Number of training batches incorporated into running statistics.
batchNorm	Population variance normalizes a training batch; unbiased sample variance updates runningVariance (a one-element batch uses population variance).
layerNorm	Normalize the selected number of trailing axes independently for each remaining prefix index.
instanceNorm	Normalize spatial axes separately per batch/channel in N,C,spatial layout.
groupNorm	Normalize channels/spatial positions per batch and channel group; channel count must be divisible by groups.
rmsNorm	Divide trailing-axis groups by sqrt(mean(x*x)+epsilon), without subtracting a mean.
weightNorm	Normalize each selected-axis slice and multiply it by the matching explicit magnitude; zero directions fail.
Initializer	Weight initializer selector: XavierKaiming uniformnormal,Orthogonal gain,or TruncatedNormal mean sigma cutoff. initializeWeights also takes explicit fan counts and RNG.
XavierUniform	Declaration and type contract below.
XavierNormal	Declaration and type contract below.
KaimingUniform	Declaration and type contract below.
KaimingNormal	Declaration and type contract below.
Orthogonal	Declaration and type contract below.
TruncatedNormal	Declaration and type contract below.
initializeWeights	Generate an immutable tensor and advanced RNG.
embedding	Gather checked row indices from [vocabulary,features] weights in requested order.
dense	Apply x*transpose(weights)+bias to [batch,in] inputs and [out,in] weights.
denseAD	Symbolic dense layer with gradients through inputs, weights, and explicit bias expressions.
ConvOptions	Per-axis stride,padding,dilation,and channel-group settings for checked convolution.
ConvOptions	Declaration and type contract below.
convolutionStride	Positive spatial stride per convolution axis.
convolutionPadding	Nonnegative symmetric padding per spatial axis.
convolutionDilation	Positive spacing between kernel elements per spatial axis.
convolutionGroups	Positive group count dividing input/output channels for grouped convolution.
convolution	Direct checked 1D2D3D cross-correlation with grouped channels and explicit stridepaddingdilation.
transposedConvolution	Direct adjoint-layout convolution; output padding is per-axis and strictly less than stride.
depthwiseConvolution	Set groups to input channels; output channel multipliers remain explicit in the weight shape.
convolutionAD	Construct the same direct convolution from scalar graph arithmetic, with automatic inputkernelbias gradients.
PoolMode	Pooling reduction selector: MaxPool or AveragePool over each checked window.
MaxPool	Declaration and type contract below.
AveragePool	Declaration and type contract below.
pooling	Windowed max/average pooling.
adaptivePooling	Partition spatial axes by floor/ceil output-bin boundaries, then reduce each bin.
globalPooling	Reduce all spatial positions to one value per N,C pair.
scaledDotProductAttention	Return attended values and normalized weights for Q[N,D],K[M,D],V[M,F]; True masks permit keys.
multiheadAttention	Split feature dimension into equal heads, attend independently, concatenate.
causalMask	Build a query-by-key allowed mask, with an explicit cached-prefix offset.
rotaryEmbedding	Rotate adjacent feature pairs at sequence positions offset+i; feature dimension must be even.
sinusoidalEncoding	Return [position,feature] sine/cosine positional encoding with base 10000.
alibiBias	Return nonpositive slope-scaled absolute position offsets; combine with a causal allowed mask if required.
KVCache	Opaque retained attention keys/values for incremental decoding; use checked cache append/attention operations.
emptyCache	Create an owned immutable key/value cache with fixed positive capacity and feature widths.
appendCache	Append equal-length key/value sequences without truncating history; reject capacity or feature mismatches.
cacheValues	Return immutable accumulated key/value tensors, preserving insertion order.
transformerFeedForward	Compose dense, checked elementwise activation, and dense layers with explicit parameters.
layerNormAD	Symbolic trailing-axis layer normalization.
rmsNormAD	Symbolic RMS normalization; its denominator includes all trailing axes.
scaledDotProductAttentionAD	First-order reverse attention graph, including Q/K score and V gradients.

Documentation

Training

data Training Source #

Layer mode: Training updates/uses batch statistics; Inference uses retained running statistics.

Constructors

Training
Inference

Instances

Instances details

Show Training Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> Training -> ShowS # show :: Training -> String # showList :: [Training] -> ShowS #
Eq Training Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: Training -> Training -> Bool # (/=) :: Training -> Training -> Bool #

dropout

dropout :: Training -> Double -> RNG -> Tensor Double -> Either MathError (Tensor Double, Tensor Double, RNG) Source #

Inverted dropout. Inference consumes no random numbers. Return a separate multiplicative mask so its exact graph can be replayed by backpropagation.

spatialDropout

spatialDropout :: Training -> Double -> RNG -> Tensor Double -> Either MathError (Tensor Double, Tensor Double, RNG) Source #

Sample one inverted dropout mask per N,C pair and share it across every spatial position.

BatchNormState

data BatchNormState Source #

Per-channel running mean/variance and incorporated training-batch count for batch normalization.

Constructors

BatchNormState
Fields runningMean :: [Double] Per-channel running mean used during inference. runningVariance :: [Double] Per-channel running variance used during inference. batchesSeen :: Int Number of training batches incorporated into running statistics.

Instances

Instances details

Show BatchNormState Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> BatchNormState -> ShowS # show :: BatchNormState -> String # showList :: [BatchNormState] -> ShowS #
Eq BatchNormState Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: BatchNormState -> BatchNormState -> Bool # (/=) :: BatchNormState -> BatchNormState -> Bool #

batchNorm

batchNorm :: Training -> Double -> Double -> [Double] -> [Double] -> BatchNormState -> Tensor Double -> Either MathError (Tensor Double, BatchNormState) Source #

Population variance normalizes a training batch; unbiased sample variance updates runningVariance (a one-element batch uses population variance).

layerNorm

layerNorm :: Int -> Double -> Tensor Double -> Either MathError (Tensor Double) Source #

Normalize the selected number of trailing axes independently for each remaining prefix index.

instanceNorm

instanceNorm :: Double -> Tensor Double -> Either MathError (Tensor Double) Source #

Normalize spatial axes separately per batch/channel in N,C,spatial layout.

groupNorm

groupNorm :: Int -> Double -> Tensor Double -> Either MathError (Tensor Double) Source #

Normalize channels/spatial positions per batch and channel group; channel count must be divisible by groups.

rmsNorm

rmsNorm :: Int -> Double -> Tensor Double -> Either MathError (Tensor Double) Source #

Divide trailing-axis groups by sqrt(mean(x*x)+epsilon), without subtracting a mean.

weightNorm

weightNorm :: Int -> [Double] -> Tensor Double -> Either MathError (Tensor Double) Source #

Normalize each selected-axis slice and multiply it by the matching explicit magnitude; zero directions fail.

Initializer

data Initializer Source #

Weight initializer selector: XavierKaiming uniformnormal,Orthogonal gain,or TruncatedNormal mean sigma cutoff. initializeWeights also takes explicit fan counts and RNG.

Constructors

XavierUniform
XavierNormal
KaimingUniform
KaimingNormal
Orthogonal Double
TruncatedNormal Double Double Double

Instances

Instances details

Show Initializer Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> Initializer -> ShowS # show :: Initializer -> String # showList :: [Initializer] -> ShowS #
Eq Initializer Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: Initializer -> Initializer -> Bool # (/=) :: Initializer -> Initializer -> Bool #

initializeWeights

initializeWeights :: Initializer -> Int -> Int -> [Int] -> RNG -> Either MathError (Tensor Double, RNG) Source #

Generate an immutable tensor and advanced RNG. Fans are explicit; orthogonal initialization requires a positive matrix shape.

embedding

embedding :: Tensor Double -> [Int] -> Either MathError (Tensor Double) Source #

Gather checked row indices from [vocabulary,features] weights in requested order.

dense

dense :: Tensor Double -> Tensor Double -> [Double] -> Either MathError (Tensor Double) Source #

Apply x*transpose(weights)+bias to [batch,in] inputs and [out,in] weights.

denseAD

denseAD :: ADTensor -> ADTensor -> [Expr] -> Either MathError ADTensor Source #

Symbolic dense layer with gradients through inputs, weights, and explicit bias expressions.

ConvOptions

data ConvOptions Source #

Per-axis stride,padding,dilation,and channel-group settings for checked convolution.

Constructors

ConvOptions
Fields convolutionStride :: [Int] Positive spatial stride per convolution axis. convolutionPadding :: [Int] Nonnegative symmetric padding per spatial axis. convolutionDilation :: [Int] Positive spacing between kernel elements per spatial axis. convolutionGroups :: Int Positive group count dividing input/output channels for grouped convolution.

Instances

Instances details

Show ConvOptions Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> ConvOptions -> ShowS # show :: ConvOptions -> String # showList :: [ConvOptions] -> ShowS #
Eq ConvOptions Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: ConvOptions -> ConvOptions -> Bool # (/=) :: ConvOptions -> ConvOptions -> Bool #

convolution

convolution :: ConvOptions -> Tensor Double -> Tensor Double -> [Double] -> Either MathError (Tensor Double) Source #

Direct checked 1D2D3D cross-correlation with grouped channels and explicit stridepaddingdilation.

transposedConvolution

transposedConvolution :: [Int] -> ConvOptions -> Tensor Double -> Tensor Double -> [Double] -> Either MathError (Tensor Double) Source #

Direct adjoint-layout convolution; output padding is per-axis and strictly less than stride.

depthwiseConvolution

depthwiseConvolution :: ConvOptions -> Tensor Double -> Tensor Double -> [Double] -> Either MathError (Tensor Double) Source #

Set groups to input channels; output channel multipliers remain explicit in the weight shape.

convolutionAD

convolutionAD :: ConvOptions -> ADTensor -> ADTensor -> [Expr] -> Either MathError ADTensor Source #

Construct the same direct convolution from scalar graph arithmetic, with automatic inputkernelbias gradients.

PoolMode

data PoolMode Source #

Pooling reduction selector: MaxPool or AveragePool over each checked window.

Constructors

MaxPool
AveragePool

Instances

Instances details

Show PoolMode Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> PoolMode -> ShowS # show :: PoolMode -> String # showList :: [PoolMode] -> ShowS #
Eq PoolMode Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: PoolMode -> PoolMode -> Bool # (/=) :: PoolMode -> PoolMode -> Bool #

pooling

pooling :: PoolMode -> [Int] -> [Int] -> [Int] -> Tensor Double -> Either MathError (Tensor Double) Source #

Windowed max/average pooling. Padded elements are excluded from the average; entirely padded windows fail.

adaptivePooling

adaptivePooling :: PoolMode -> [Int] -> Tensor Double -> Either MathError (Tensor Double) Source #

Partition spatial axes by floor/ceil output-bin boundaries, then reduce each bin.

globalPooling

globalPooling :: PoolMode -> Tensor Double -> Either MathError (Tensor Double) Source #

Reduce all spatial positions to one value per N,C pair.

scaledDotProductAttention

scaledDotProductAttention :: Tensor Double -> Tensor Double -> Tensor Double -> Maybe [[Bool]] -> Maybe [[Double]] -> Either MathError (Tensor Double, Tensor Double) Source #

Return attended values and normalized weights for Q[N,D],K[M,D],V[M,F]; True masks permit keys.

multiheadAttention

multiheadAttention :: Int -> Tensor Double -> Tensor Double -> Tensor Double -> Maybe [[Bool]] -> Either MathError (Tensor Double) Source #

Split feature dimension into equal heads, attend independently, concatenate. Projection matrices are explicit dense layers composed by the caller.

causalMask

causalMask :: Int -> Int -> Int -> Either MathError [[Bool]] Source #

Build a query-by-key allowed mask, with an explicit cached-prefix offset.

rotaryEmbedding

rotaryEmbedding :: Double -> Int -> Tensor Double -> Either MathError (Tensor Double) Source #

Rotate adjacent feature pairs at sequence positions offset+i; feature dimension must be even.

sinusoidalEncoding

sinusoidalEncoding :: Int -> Int -> Int -> Either MathError (Tensor Double) Source #

Return [position,feature] sine/cosine positional encoding with base 10000.

alibiBias

alibiBias :: Double -> Int -> Int -> Int -> Either MathError [[Double]] Source #

Return nonpositive slope-scaled absolute position offsets; combine with a causal allowed mask if required.

KVCache

data KVCache Source #

Opaque retained attention keys/values for incremental decoding; use checked cache append/attention operations.

Instances

Instances details

Show KVCache Source #
Instance details Defined in HMath.DeepLearning.Layers Methods showsPrec :: Int -> KVCache -> ShowS # show :: KVCache -> String # showList :: [KVCache] -> ShowS #
Eq KVCache Source #
Instance details Defined in HMath.DeepLearning.Layers Methods (==) :: KVCache -> KVCache -> Bool # (/=) :: KVCache -> KVCache -> Bool #

emptyCache

emptyCache :: Int -> Int -> Int -> Either MathError KVCache Source #

Create an owned immutable key/value cache with fixed positive capacity and feature widths.

appendCache

appendCache :: KVCache -> Tensor Double -> Tensor Double -> Either MathError KVCache Source #

Append equal-length key/value sequences without truncating history; reject capacity or feature mismatches.

cacheValues

cacheValues :: KVCache -> (Tensor Double, Tensor Double) Source #

Return immutable accumulated key/value tensors, preserving insertion order.

transformerFeedForward

transformerFeedForward :: Activation -> Tensor Double -> Tensor Double -> [Double] -> Tensor Double -> [Double] -> Either MathError (Tensor Double) Source #

Compose dense, checked elementwise activation, and dense layers with explicit parameters.

layerNormAD

layerNormAD :: Int -> Double -> ADTensor -> Either MathError ADTensor Source #

Symbolic trailing-axis layer normalization. Affine gamma/beta can be composed with broadcast multiplication/addition in HMath.Autodiff.

rmsNormAD

rmsNormAD :: Int -> Double -> ADTensor -> Either MathError ADTensor Source #

Symbolic RMS normalization; its denominator includes all trailing axes.

scaledDotProductAttentionAD

scaledDotProductAttentionAD :: ADTensor -> ADTensor -> ADTensor -> Maybe [[Bool]] -> Either MathError ADTensor Source #

First-order reverse attention graph, including Q/K score and V gradients. Masks have the same True=allowed semantics as the checked numeric operation; a fully masked row is a constant zero graph and backpropagates exact zeros.