Title:	Self-Attention Algorithm
Version:	0.4.0
Description:	Self-Attention algorithm helper functions and demonstration vignettes of increasing depth on how to construct the Self-Attention algorithm, this is based on Vaswani et al. (2017) <doi:10.48550/arXiv.1706.03762>, Dan Jurafsky and James H. Martin (2022, ISBN:978-0131873216) https://web.stanford.edu/~jurafsky/slp3/ "Speech and Language Processing (3rd ed.)" and Alex Graves (2020) https://www.youtube.com/watch?v=AIiwuClvH6k "Attention and Memory in Deep Learning".
License:	GPL (≥ 3)
Encoding:	UTF-8
RoxygenNote:	7.2.3
Suggests:	covr, knitr, rmarkdown, testthat (≥ 3.0.0)
VignetteBuilder:	knitr
Config/testthat/edition:	3
NeedsCompilation:	no
Packaged:	2023-11-10 00:29:14 UTC; bquast
Author:	Bastiaan Quast [aut, cre]
Maintainer:	Bastiaan Quast <bquast@gmail.com>
Repository:	CRAN
Date/Publication:	2023-11-10 03:10:02 UTC

SoftMax sigmoid function

Description

SoftMax sigmoid function

Usage

ComputeWeights(scores)

Arguments

Value

output value (numeric)

Examples

# Set up a scores matrix
scores <- matrix(c( 6,  4, 10,  5,
                    4,  6, 10,  6,
                   10, 10, 20, 11,
                    3,  1,  4,  2),
                 nrow  = 4,
                 ncol  = 4,
                 byrow = TRUE)

# Compute the weights based on the scores matrix
ComputeWeights(scores)

# this outputs
#            [,1]       [,2]      [,3]       [,4]
# [1,] 0.10679806 0.03928881 0.7891368 0.06477630
# [2,] 0.03770440 0.10249120 0.7573132 0.10249120
# [3,] 0.00657627 0.00657627 0.9760050 0.01084244
# [4,] 0.27600434 0.10153632 0.4550542 0.16740510

Maximum of Matrix Rows

Description

Maximum of Matrix Rows

Usage

RowMax(x)

Arguments

Value

output value (numeric)

Examples

# generate a matrix of integers (also works for floats)
set.seed(0)
M = matrix(floor(runif(9, min=0, max=3)),
           nrow=3,
           ncol=3)
print(M)

# this outputs
#      [,1] [,2] [,3]
# [1,]    2    1    2
# [2,]    0    2    2
# [3,]    1    0    1

# apply RowMax() to the matrix M, reformat output as matrix again
# to keep the maxs on their corresponding rows
RowMax(M)

# this outputs
#      [,1]
# [1,]    2
# [2,]    2
# [3,]    1

SoftMax sigmoid function

Description

SoftMax sigmoid function

Usage

SoftMax(x)

Arguments

Value

output value (numeric)

Examples

# create a vector of integers (also works for non-integers)
set.seed(0)
V = c(floor(runif(9, min=-3, max=3)))
print(V)

# this outputs
# [1]  2 -2 -1  0  2 -2  2  2  0

# apply the SoftMax() function to V
sV <- SoftMax(V)
print(sV)

# this outputs
# [1] 0.229511038 0.004203641 0.011426682 0.031060941
# 0.229511038 0.004203641 0.229511038 0.229511038 0.031060941

Attnention mechanism

Description

Attnention mechanism

Usage

attention(Q, K, V, mask = NULL)

Arguments

Value

attention values