Convex Functions

Convex Optimization > Convex sets | Convex functions | Convex optimization problems | Algorithms | DCP

Definition
Alternate characterizations of convex functions
Operations that preserve convexity:
- Composition with an affine function
- Point-wise maximum
- Nonnegative weighted sum
- Partial minimum
- Composition with monotone convex functions

Definition

Domain of a function

The domain of a function $f: mathbf{R}^n rightarrow mathbf{R}$ is the set $mbox{bf dom} f subseteq mathbf{R}^n$ over which is well-defined, in other words:

$mbox{bf dom} f :={ x in mathbf{R}^n: -infty < f(x) < +infty}.$

Here are some examples:

The function with values has domain $mbox{bf dom} f=mathbf{R}_{++}$ .
The function with values has domain $mbox{bf dom} f={cal S}_{++}^n$ (the set of positive-definite matrices).

Definition of convexity

A function $f : mathbf{R}^n rightarrow mathbf{R}$ is convex if

is convex;
In addition,

$forall, x_1,x_2 in mbox{bf dom} f, ;; forall, theta in [0, 1], f left(theta x + (1-theta) yright) le theta f(x) + (1-theta) f(y) .$

Note that the convexity of the domain is required. For example, the function f : mathbf{R} rightarrow mathbf{R} defined as

f(x) = left{ begin{array}{ll} x & mbox{if } x notin [-1,1] +infty & mbox{otherwise} end{array}right.

is not convex, although is it linear (hence, convex) on its domain ]-infty,-1)cup(1,+infty[ .

We say that a function is concave if is convex.

Examples:

The function defined as for and for is convex. However, the function with domain the whole line except is not, since that domain is not convex.

Alternate characterizations of convexity

Let $f : mathbf{R}^n rightarrow mathbf{R}$ . The following are equivalent conditions for to be convex.

Epigraph

A function $f : mathbf{R}^n rightarrow mathbf{R}$ is convex if and only if its epigraph

$mbox{bf epi} f := left{ (x,t) in mathbf{R}^{n+1} ~:~ t ge f(x) right}$

is convex.

Example: We can us this result to prove for example, that the largest eigenvalue function $lambda_{rm max} : {cal S}^n rightarrow mathbf{R}$ , which to a given n times n symmetric matrix associates its largest eigenvalue, is convex, since the condition $lambda_{rm max}(X) le t$ is equivalent to the condition that $t I - X in {cal S}_+^n$ .

First-order condition

If is differentiable (that is, mbox{bf dom} f is open and the gradient exists everywhere on the domain), then is convex if and only if

forall : x,y ~:~ f(y) ge f(x) + nabla f (x)^T(y-x) .

The geometric interpretation is that the graph of is bounded below everywhere by anyone of its tangents.

Restriction to a line

The function is convex if and only if its restriction to /any} line is convex, meaning that for every $x_0 in mathbf{R}^n$ , and $v in mathbf{R}^n$ , the function g(t) := f(x_0+tv) is convex.

Examples:

Convexity of the log-determinant function.

Second-order condition

If is twice differentiable, then it is convex if and only if its Hessian nabla^2 f is positive semi-definite everywhere on the domain of . This is perhaps the most commonly known characterization of convexity.

Examples:

Operations that preserve convexity

Composition with an affine function

The composition with an affine function preserves convexity: if $A in mathbf{R}^{m times n}$ , $b in mathbf{R}^m$ and $f : mathbf{R}^m rightarrow mathbf{R}$ is convex, then the function $g : mathbf{R}^n rightarrow mathbf{R}$ with values g(x) = f(Ax+b) is convex.

Point-wise maximum

The pointwise maximum of a family of convex functions is convex: if $(f_alpha)_{alpha in {cal A}}$ is a family of convex functions index by alpha , then the function

$f(x) := max_{alpha in {cal A}} : f_alpha(x)$

is convex. This is one of the most powerful ways to prove convexity.

Examples:

Dual norm: for a given norm, we define the dual norm as the function

$x rightarrow max_{y :: |y| le 1} : y^Tx .$

This function is convex, as the maximum of convex (in fact, linear) functions (indexed by the vector ). The dual norm earns its name, as it satisfies the properties of a norm.

Largest singular value of a matrix: Another example is the largest singular value (see also here) of a matrix :

$f(A) = sigma_{rm max}(A) = displaystylemax_{x::: |x|_2 = 1} |Ax|_2.$

Here, each function (indexed by $x in mathbf{R}^n$ ) A rightarrow |Ax|_2 is convex, since it is the composition of the Euclidean norm (a convex function) with an affine function A rightarrow Ax .

Nonnegative weighted sum

The nonnegative weighted sum of convex functions is convex.

Example: Negative entropy function.

Partial minimum

If is a convex function in x=(y,z) , then the function g(y) := min_z : f(y,z) is convex. (Note that joint convexity in (y,z) is essential.)

Example:

Case when is quadratic: the Schur complement lemma.

Composition with monotone convex functions

The composition with another function does not always preserve convexity. However, if f = h circ g , with h,g convex and increasing, then is convex.

Indeed, the condition f(x) le t is equivalent to the existence of such that

The condition above defines a convex set in the space of (x,y,t) -variables. The epigraph of is thus the projection of that convex set on the space of (x,t) -variables, hence it is convex.

Example: proving convexity via monotonicity.

More generally, if the functions $g_i : mathbf{R}^n rightarrow mathbf{R}$ , i=1,ldots,k are convex and $h : mathbf{R}^k rightarrow mathbf{R}$ is convex and non-decreasing in each argument, with $mbox{bf dom}g_i = mbox{bf dom} h = mathbf{R}$ , then x rightarrow (h circ g)(x) := h(g_1(x),ldots,g_k(x)) is convex.

For example, if g_i 's are convex, then log sum_i exp g_i also is.