C++ Neural Networks and Fuzzy Logic by Valluru B. Rao M&T Books, IDG Books Worldwide, Inc. ISBN: 1558515526   Pub Date: 06/01/95

Appendix B
Mathematical Background

Dot Product or Scalar Product of Two Vectors

Given vectors U and V, where U = (u₁, …, u_n) and V = (v₁, …, v_n), their dot product or scalar product is U • V = u₁v₁ +… + u_nv_n = Σ u_i v_i.

Matrices and Some Arithmetic Operations on Matrices

A real matrix is a rectangular array of real numbers. A matrix with m rows and n columns is referred to as an mxn matrix. The element in the ith row and jth column of the matrix is referred to as the ij element of the matrix and is denoted by a_ij.

The transpose of a matrix M is denoted by M^T. The element in the ith row and jth column of M^T is the same as the element of M in its jth row and ith column. M^T is obtained from M by interchanging the rows and columns of M. For example, if

            2  7 -3              2  4
       M =          , then MT =  7  0
            4  0  9             -3  9

If X is a vector with m components, x₁, …, x_m, then it can be written as a column vector with components listed one below another. It can be written as a row vector, X = (x₁, …, x_m). The transpose of a row vector is the column vector with the same components, and the transpose of a column vector is the corresponding row vector.

The addition of matrices is possible if they have the same size, that is, the same number of rows and same number of columns. Then you just add the ij elements of the two matrices to get the ij elements of the sum matrix. For example,

       3 -4 5    5 2 -3      8 -2  2
               +         =
       2  3 7    6 0  4      8  3 11

Multiplication is defined for a given pair of matrices, only if a condition on their respective sizes is satisfied. Then too, it is not a commutative operation. This means that if you exchange the matrix on the left with the matrix on the right, the multiplication operation may not be defined, and even if it is, the result may not be the same as before such an exchange.

The condition to be satisfied for multiplying the matrices A, B as AB is, that the number of columns in A is equal to the number of rows in B. Then to get the ij element of the product matrix AB, you take the ith row of A as one vector and the jth column of B as a second vector and do a dot product of the two. For example, the two matrices given previously to illustrate the addition of two matrices are not compatible for multiplication in whichever order you take them. It is because there are three columns in each, which is different from the number of rows, which is 2 in each. Another example is given as follows.

                 3 -4 5              5  6
       Let A =            and   B =  2  0
                 2  3 7             -3  4

Then AB and BA are both defined, AB is a 2x2 matrix, whereas BA is 3x3.

                 -8  38              27 -2 67
       Also AB =         and BA  =    6 -8 10
                 -5  40              -1 24 13

Lyapunov Function

A Lyapunov function is a function that decreases with time, taking on non-negative values. It is used to correspond between the state variables of a system and real numbers. The state of the system changes as time changes, and the function decreases. Thus, the Lyapunov function decreases with each change of state of the system.

We can construct a simple example of a function with the property of decreasing with each change of state as follows. Suppose a real number, x, represents the state of a dynamic system at time t. Also suppose that x is bounded for any t by a positive real number M. That means x is less than M for every value of t.

Then the function,

       f(x,t) = exp(-|x|/(M+|x|+t))

is non-negative and decreases with increasing t.

Local Minimum

A function f(x) is defined to have a local minimum at y, with a value z, if

       f(y) = z, and f(x) ≥ z, for each x, such that there exists
       a positive real number h such that y – h ≤ x ≤ y + h.

In other words, there is no other value of x in a neighborhood of y, where the value of the function is smaller than z.

There can be more than one local minimum for a function in its domain. A Step function (with a graph resembling a staircase) is a simple example of a function with an infinite number of points in its domain with local minima.

Global Minimum

A function f(x) is defined to have a global minimum at y, with a value z, if

       f(y) = z, and f(x) ≥ z, for each x in the domain of
       the function f.

In other words, there is no other value of x in the domain of the function f, where the value of the function is smaller than z. Clearly, a global minimum is also a local minimum, but a local minimum may not be a global minimum.

There can be more than one global minimum for a function in its domain. The trigonometric function f(x) = sinx is a simple example of a function with an infinite number of points with global minima. You may recall that sin(3π/ 2), sin (7π/ 2), and so on are all –1, the smallest value for the sine function.

Kronecker Delta Function

The Kronecker delta function is a function of two variables. It has a value of 1 if the two arguments are equal, and 0 if they are not. Formally,

                       1 if x = y
       δ(x,y)=
                       0 if x ≠ y

Gaussian Density Distribution

The Gaussian Density distribution, also called the Normal distribution, has a density function of the following form. There is a constant parameter c, which can have any positive value.