SJSU Singular Matrix Database

Rank as used in theoretical mathematics and numerical rank

In theoretical mathematics a singular matrix is usually defined as an n by n matrix with rank less than n. For rectangular matrices it is natural to generalize this and to define an m by n matrix to be singular if its rank is less than min(m,n) . However, these definitions have difficulties in the presence of errors in the matrix elements. To quote from Matrix Computations, by Gene Golub and Charles Van Loan, third edition, page 72:

Numerous theorems in linear algebra have the form "if such-and-such a matrix has full rank, then such-and-such property holds." While neat and aesthetic, results of this flavor do not help us address the numerical difficulties frequently encountered in situations where near rank deficiency prevails. Rounding errors and fuzzy data make rank determination a nontrivial exercise. Indeed, for some small ε we may be interested in the ε rank of a matrix which we define by
rank(A,ε) = min ||A-B|| ≤ ε rank(B).
Thus if A is obtained in a laboratory with each aij correct to within ± .001 then it might make sense to look at rank(A, .001). Along these same lines, if A is an m by n floating point matrix then it is reasonable to regard A as numerically rank deficient if rank(A,ε) < min(m,n) with ε = u ||A||.
In this quote Golub and Van Loan are using the letter u to indicate relative machine precision and the norm ||   || to indicate the Euclidean or two norm. Golub and Van Loan prove that rank(A, tol) can be calculated by determining the number of singular values greater than tol. Therefore our use of the term "numerically singular" is the same as Golub and Van Loan's use of "numerically rank deficient" except that our tolerance for the numerical rank is slightly different. We use a tolerance of max(m,n) eps( norm(A) ) which is also the default tolerance in Matlab's rank command.

The difference between exact rank and numerical rank can also be characterized in terms of matrix null spaces. An m by n matrix A has rank r if the dimension of the null space of A is n-r. This means that there exist a subspace S of dimension n-r such that

A x = 0   for x ∈ S          (1)
and (1) is true for no subspace of dimension greater than n-r.

In practice, in a floating point number system, due to computer arithmetic errors the calculated value of A x in (1) will usually not be exactly zero for x ≠ 0. Therefore, in practice, one must be satisfied if A x in (1) is "almost zero" or "numerically zero." The concept of numerical rank captures this idea in a precise sense. Using the minimax characterization of singular values one can show that if A has numerical rank r with tolerance tol then there exists a subspace S of dimension n-r such that

max x ≠ 0 ||A x || / || x || ≤ tol   for x ∈ S           (2)
and that (2) is true for no subspace of dimension greater than n-r. Therefore if we select the tolerance tol to be a reasonable estimate or bound on the computer arithmetic errors in evaluating ||A x || / || x ||, then (2) is the natural extension of (1) when the errors in A are due to computer arithmetic.

For an example see this link .