Matrix Products

Matrix-vector product
Matrix-matrix product
Block matrix product
Trace and scalar product

Matrix-vector product

Definition

We define the matrix-vector product between a m times n matrix and a -vector , and denote by , the -vector with -th component

$(Ax)_i = sum_{j=1}^n A_{ij}x_j , ;; i=1,ldots,m.$

The picture on the left shows a symbolic example with n=2 and m=3 . We have y = Ax , that is:

$begin{array}{rcl} y_mathbf{1} &=& A_{mathbf{1}1}x_1 + A_{mathbf{1}2}x_2 , y_mathbf{2} &=& A_{mathbf{2}1}x_1 + A_{mathbf{2}2}x_2 , y_mathbf{3} &=& A_{mathbf{3}1}x_1 + A_{mathbf{3}2}x_2 . end{array}$

Interpretation as linear combinations of columns

If the columns of are given by the vectors a_i , i=1,ldots,n , so that A = (a_1 , ldots, a_n) , then can be interpreted as a linear combination of these columns, with weights given by the vector :

$Ax = sum_{i=1}^n x_i a_i .$

In the above symbolic example, we have y = Ax , that is:

$y = x_mathbf{1} left(begin{array}{c} A_{1mathbf{1}} A_{2mathbf{1}} A_{3mathbf{1}} end{array}right) + x_mathbf{2} left(begin{array}{c} A_{1mathbf{2}} A_{2mathbf{2}} A_{3mathbf{2}} end{array}right) .$

Example:

Interpretation as scalar products with rows

Alternatively, if the rows of are the row vectors a_i^T , i=1,ldots,m :

$A = left( begin{array}{c} a_1^T vdots a_m^T end{array} right),$

then is the vector with elements a_i^Tx , i=1,ldots,m :

$Ax = left( begin{array}{c} a_1^Tx vdots a_m^Tx end{array} right) .$

In the above symbolic example, we have y = Ax , that is:

$begin{array}{rcl} y_mathbf{1} &=& left(begin{array}{c} A_{mathbf{1}1} A_{mathbf{1}2} end{array}right)^Tleft(begin{array}{c} x_1 x_2 end{array}right) = A_{mathbf{1}1} x_1 + A_{mathbf{1}2}x_2, y_mathbf{2} &=& left(begin{array}{c} A_{mathbf{2}1} A_{mathbf{2}2} end{array}right)^Tleft(begin{array}{c} x_1 x_2 end{array}right) = A_{mathbf{2}1}x_1 + A_{mathbf{2}2}x_2, y_mathbf{3} &=& left(begin{array}{c} A_{mathbf{3}1} A_{mathbf{3}2} end{array}right)^Tleft(begin{array}{c} x_1 x_2 end{array}right) = A_{mathbf{3}1}x_1 + A_{mathbf{3}2}x_2. end{array}$

Example: Absorption spectrometry: using measurements at different light frequencies.

Left product

If $z in mathbf{R}^m$ , then the notation z^TA is the row vector of size equal to the transpose of the column vector $A^Tz in mathbf{R}^n$ . That is:

$(z^TA)_j = sum_{i=1}^m A_{ij}z_i , ;; j=1,ldots,n.$

Example: Return to the network example, involving a m times n incidence matrix. We note that, by construction, the columns of sum to zero, which can be compactly written as $mathbf{1}^T A = 0$ , or $A^Tmathbf{1} = 0$ .

Matlab syntax

The product operator in Matlab is *. If the sizes are not consistent, Matlab will produce an error.

Matlab syntax
>> A = [1 2; 3 4; 5 6]; % 3x2 matrix
>> x = [-1; 1]; % 2x1 vector
>> y = A*x; % result is a 3x1 vector
>> z = [-1; 0; 1]; % 3x1 vector
>> y = z'*A; % result is a 1x2 (i.e., row) vector

Matrix-matrix product

Definition

We can extend matrix-vector product to matrix-matrix product, as follows. If $A in mathbf{R}^{m times n}$ and $B in mathbf{R}^{n times p}$ , the notation denotes the m times p matrix with i,j element given by

$(AB)_{ij} = sum_{k=1}^n A_{ik} B_{kj} .$

Transposing a product changes the order, so that (AB)^T = B^TA^T .

Column-wise interpretation

If the columns of are given by the vectors b_i , i=1,ldots,n , so that B = [b_1 , ldots, b_n] , then can be written as

$AB = A left( begin{array}{ccc} b_1 & ldots & b_n end{array} right) = left( begin{array}{ccc} Ab_1 & ldots & Ab_n end{array} right) .$

In other words, results from transforming each column b_i of into Ab_i .

Row-wise interpretation

The matrix-matrix product can also be interpreted as an operation on the rows of . Indeed, if is given by its rows a_i^T , i=1,ldots,m , then is the matrix obtained by transforming each one of these rows via , into a_i^TB , :

$AB = left( begin{array}{c} a_1^T vdots a_n^T end{array} right) B = left( begin{array}{c} a_1^TB vdots a_n^TB end{array} right) .$

(Note that a_i^TB 's are indeed row vectors, according to our matrix-vector rules.)

Block Matrix Products

Matrix algebra generalizes to blocks, provided block sizes are consistent. To illustrate this, consider the matrix-vector product between a m times n matrix and a -vector , where A,x are partitioned in blocks, as follows:

$A = left(begin{array}{cc} A_{1} & A_{2} end{array} right), ;; x = left(begin{array}{c} x_1 x_2 end{array}right),$

where A_i is m times n_i , $x_i in mathbf{R}^{n_i}$ , i=1,2 , n_1+n_2=n . Then

Symbolically, it's as if we would form the ‘‘scalar’’ product between the ‘‘row vector (A_1,A_2) and the column vector (x_1,x_2) !

Likewise, if a n times p matrix is partitioned into two blocks B_i , each of size n_i , i=1,2 , with n_1+n_2=n , then

$AB = left(begin{array}{cc} A_{1} & A_{2} end{array} right) left(begin{array}{c} B_1 B_2 end{array}right) = A_1B_1+A_2B_2.$

Again, symbolically we apply the same rules as for the scalar product — except that now the result is a matrix.

Example: Gram matrix.

Finally, we can consider so-called outer products. Consider the case for example when is a m times n matrix partitioned row-wise into two blocks A_1,A_2 , and is a n times p matrix that is partitioned column-wise into two blocks B_1,B_2 :

$A = left(begin{array}{c} A_{1} A_{2} end{array} right) , ;; B = left(begin{array}{cc} B_{1} & B_{2} end{array} right).$

Then the product C=AB can be expressed in terms of the blocks, as follows:

$C = AB = left(begin{array}{c} A_{1} A_{2} end{array} right) left(begin{array}{cc} B_{1} & B_{2} end{array} right) = left(begin{array}{cc} A_1B_{1} & A_1B_{2} A_2B_{1} & A_2B_{2} end{array} right).$

Trace, scalar product

Trace

The trace of a square n times n matrix , denoted by mbox{bf Tr} A , is the sum of its diagonal elements: $mbox{bf Tr} A = sum_{i=1}^n A_{ii}$ .

Some important properties:

Trace of transpose: The trace of a square matrix is equal to that of its transpose.
Commutativity under trace: for any two matrices $A in mathbf{R}^{m times n}$ and $B in mathbf{R}^{n times m}$ , we have

Matlab syntax
>> A = [1 2 3; 4 5 6; 7 8 9]; % 3x3 matrix
>> tr = trace(A); % trace of A

Scalar product between matrices

We can define the scalar product between two m times n matrices A,B via

$langle A,B rangle = mbox{bf Tr} (A^TB) = sum_{i=1}^m sum_{j=1}^m A_{ij}B_{ij}.$

The above definition is symmetric: we have

$langle A,B rangle = mbox{bf Tr} (A^TB) = mbox{bf Tr} (A^TB)^T = mbox{bf Tr} (B^TA) = langle B,A rangle.$

Our notation is consistent with the definition of the scalar product between two vectors, where we simply view a vector in $mathbf{R}^n$ as a matrix in $mathbf{R}^{n times 1}$ . We can interpret the matrix scalar product as the vector scalar product between two long vectors of length each, obtained by stacking all the columns of A,B on top of each other.

Matlab syntax
>> A = [1 2; 3 4; 5 6]; % 3x2 matrix
>> B = randn(3,2); % random 3x2 matrix
>> scal_prod = trace(A'*B); % scalar product between A and B
>> scal_prod = A(:)'*B(:); % this is the same as the scalar product between the
                            % vectorized forms of A, B