Convergence

You can read the LaTeX document online (for the latest updated chapters) from the link: probability.pdf

Chapter 5: Convergence

Contents
Contents
 1.  Convergence of random variables
 2.  Convergence of measures
 3.  Relationships among various modes of convergence

1. Convergence of random variables Here and hereafter the term ''convergence'' will be used to mean convergence to a finite limit. The first notion of convergence would be the following one:

The proof can be found in this link: https://gaomj.cn/convergence-almost-surely/.

A weaker concept of convergence is of basic importance in probability theory.

From the previous theorem we have the immediate consequence below.

Like in functional analysis, we mostly consider the case where 1\leqslant p\leqslant\infty.

We will investigate the relation of these modes of convergence in some later sections. First let us take a look at the following proposition. It can be a corollary of some later results.

This section ends with another kind of convergence which is basic in functional analysis. However we confine ourselves to L^1 here. Recall in functional analysis, when X is a normed vector space, \{x _ n\}\subseteq X is said to converge weakly in X if there exists x\in X such that for each x'\in X'(the dual space), x'(x _ n)\to x'(x) as n\to\infty. In L^1, given any random variable that is bounded a.e., the relation \ell(X)=\mathbb E(XY) defines a continuous linear functional \ell on L^1. Furthermore, the linear isometry Y\to\ell defined in this fashion is bijective, i.e., given any continuous linear functional \ell on L^1, there exists one and only one Y which is bounded a.e. such that \ell(X)=\mathbb E(XY) for all X\in L^1. This result can be named F. Riesz representation theorem in L^1.

Hence \{X _ n\} in L^1 is said to converge weakly in L^1 to X iff for each Y that is bounded a.e. we have \mathbb E(X _ nY)=E(XY). Clearly convergence in L^1 defined above implies weak convergence; hence the former is sometimes referred to as ''strong''.

Borel-Cantelli lemma
The Borel–Cantelli lemma is a theorem about sequences of events. The following lemma is usually called the first Borel-Cantelli lemma.

It can be seen easily that if \sum _ {n=1}^\infty\mathbb{P}(|X _ n-X|>\varepsilon)<\infty for all \varepsilon>0, then X _ n\to X almost everywhere, by the following theorem:

This theorem can be shown by
\[
\lim _ {k\to\infty}\mathbb{P}\Big(\bigcap _ {n=k}^\infty\, \{|X _ n-X|<\varepsilon\}\Big)=\lim _ {k\to\infty} \mathbb{P}(|X _ n-X|<\varepsilon,\, \forall\, n\geqslant k)=1.
\]

The proof is the same as the one in real analysis.

Under the assumption of independence, the first Borel-Cantelli lemma has a striking complement.

For the proof, we just need \mathbb P(\liminf A _ n^c)=0, i.e., \mathbb P(\bigcap _ {n=m}^\infty A _ n^c)\to0\, (m\to\infty). From the independence we have \mathbb P(\bigcap _ {n=m}^{m'} A _ n^c)=\prod _ {n=m}^{m'}(1-\mathbb P(A _ n))\leqslant\exp(-\sum _ {n=m}^{m'}\mathbb P(A _ n))\to0 as m'\to\infty. For the case where the events are pairwise independent, the proof is a little longer. Let I _ n:=\chi _ {A _ n}, so that the hypothesis of pairwise independence becomes \mathbb E(I _ mI _ n)=\mathbb E(I _ m)\mathbb E(I _ n) for any different m,n; the conclusion we desired becomes \mathbb P(\sum I _ n=+\infty)=1, because \omega\in \limsup A _ n just means \omega was in an infinite number of A _ n; and the hypothesis \sum\mathbb P(A _ n)=\infty becomes \sum\mathbb E(I _ n)=+\infty.

Consider the partial sum J _ k=\sum _ {n=1}^{k}I _ n. Using Chebyshev's inequality, we have for every A>0, \mathbb P(|J _ k-\mathbb E(J _ k)|\leqslant A\sigma(J _ k))\geqslant1-\frac{\mathrm{Var}(J _ k)}{A^2\mathrm{Var}(J _ k)}=1-\frac1{A^2}. Denote p _ n=\mathbb E(I _ n). Now we want to calculate \mathrm{Var}(J _ k)=\mathbb E(J _ k^2)-\mathbb E(J _ k)^2. We have \mathbb E(J _ k^2)=\sum \mathbb E(I _ n^2)+2\sum \mathbb E(I _ m)E(I _ n)=\sum p _ n^2+2\sum p _ mp _ n+\sum(p _ n-p _ n^2), so \mathrm {Var}(J _ k)=\sum _ {i=1}^{k}(p _ n-p _ n^2). If you were familiar with the notion of covariance the calculation might be much simplified. Now since \sum p _ n=\mathbb E(J _ k)\to\infty, it follows that \sigma(J _ k)\leqslant(\sum p _ n)^{1/2}=\sqrt{\mathbb E(J _ k)}=o(\mathbb E(J _ k)). Note that we have obtained before that with probability at least 1-\frac1{A^2} it holds that |J _ k-\mathbb E(J _ k)|\leqslant A\sigma(J _ k). For sufficiently large k the right side is less than \frac12 \mathbb E(J _ k), so with probability at least 1-\frac1{A^2} we have J _ k\geqslant\frac12\mathbb E(J _ k), and therefore \lim _ n J _ n\geqslant\frac12\mathbb E(J _ k) (J _ k is increasing). Thus letting k\to\infty we obtain it holds that \lim _ n J _ n=+\infty with probability at least 1-\frac1{A^2}. Since A can be arbitrary large, \lim _ n J _ n=+\infty a.e.

The first Borel-Cantelli lemma is often referred to as the ''convergence part'' and the second ''the divergence part''. The first is more useful since the events there may be completely arbitrary. The second has the extension to pairwise independent random variables; although the result is of some interest, it is the method of proof given that is more important. It is a useful technique in probability theory.

2. Convergence of measures In this section the contents are provided in a general framework. All the proofs are not given because maybe they are beyond the scope of this review.

Due to the generality, we need some definitions and facts from topology. For example, the sets that are compact, relatively compact, sequentially compact (relatively sequentially compact), locally compact (any point has an open neighborhood whose closure is compact), may be considered. The concept of Polish space is mentioned in the previous chapters. In short, a Polish space is a separable completely metrizable topological space. This is one type of topological space that is of particular importance in probability theory. Examples of Polish spaces are countable discrete spaces, the Euclidean spaces \mathbb R^n, and the space C[0, 1] of continuous functions, equipped with the sup-norm. In practice, all spaces that are of importance in probability theory are Polish spaces.

In the following, let (E, \tau) be a topological space with Borel \sigma-algebra \mathcal E=\mathcal B(E):=\sigma(\tau) and with complete metric d. Define the following spaces of measures on E: \mathcal M _ 1(E):=\{\text{probability measures on }E\}, and \mathcal M _ {\leqslant1}(E):=\{\text{finite measures on }E\mid \mu(E)\leqslant 1\}. The elements in the latter are called sub-probability measures on E. Define the following spaces of continuous functions on E: let C(E) be all continuous functions on E, C _ K(E) be all continuous functions with compact support, C _ 0(E) be all continuous functions that vanish at infinity, and C _ b(E) be all bounded continuous functions.

Now we are ready for the definitions of weak convergence and vague convergence.

Weak convergence is a notion basically introduced in functional analysis. If we consider the set of all finite signed measures, denoted as X. It can be shown that X can be equipped with a so-called total variation norm |\cdot|: \mu\mapsto\sup _ A(\mu(A)-\mu(A^c)). For this normed vector space, we consider the dual space X' of continuous linear functionals X\to\mathbb R. The weak convergence in functional analysis under the topology induced by this norm is: \mu _ n\stackrel{w}{\to}\mu if x'(\mu _ n)\to x'(\mu) for every x'\in X'. This can be proved to be equivalent to: (\mu _ n) is bounded and \mu _ n(A)\to\mu(A) for any measurable set A. However, this is slightly different from the convergence in Definition 14, which holds if and only if \mu _ n(A)\to\mu(A) for all measurable set A with \mu(\partial A)=0 (this kind of A is called continuity set). Hence this is weaker than the one in functional analysis. In short, if we topologize in the above way and call the convergence in Definition 14 ''weak convergence'', then this kind of weak convergence is somewhat different from the one in functional analysis.

If we topologize in another way, we can see weak convergence induces on \mathcal M _ f(E) actually the weak* topology. It is believed that this brings the main seasons why some old books prefer to use ''vague convergence'' rather than ''weak convergence''. The authors would find the conflicts on terminology of ''weak'' between probability and functional analysis! They actually got ''weak *'' instead of ''weak'' in the context of functional analysis. People seldom use ''weak * convergence'' in probability but ''vague convergence'' or ''weak convergence'' because they don't like saying something like ''converge weak-star-ly'' or putting a lot of asterisks in their texts.

Recall that in functional analysis, when X is a normed vector space, (x _ n')\subseteq X' is said to weakly * converge in X' if there exists x'\in X' such that for each x\in X, x' _ n(x)\to x'(x), and the weak * topology on X is by definition the weakest topology on X such that all the mappings \varphi _ x:x'\in X'\mapsto \varphi(x'):= x'(x)\in\mathbb K, are continuous. In the case here, X=C _ b(E) and the norm is the sup-norm |\cdot| _ \infty. The dual is all functionals that map a bounded continuous function f on E to a number in \mathbb K. Consider the functionals: f\in C _ b(E)\mapsto \mu(f):=\int f\, \mathrm d\mu. Obviously they are all linear and continuous, so \mathcal M _ f(E)\subseteq C _ b(E)'. By the definition of weak * convergence in functional analysis, a sequence (\mu _ n) is said to weakly * converge in C _ b(E)' if there exists \mu\in C _ b(E)' such that \mu _ n(f)\to\mu(f) for all f\in C _ b(E), which coincides with the defining formula in Definition 14. \mathcal M _ f(E)\subseteq C _ b(E)' implies that the induced topology here is the trace of the weak * topology on \mathcal M _ f(E) (trace topology is also called subspace topology).

The contents of the following theorem can be called Portemanteau theorem.

Prohorov's theorem
In the following, let E be a Polish space with Borel \sigma-algebra \mathcal E. A fundamental question is: When does a sequence \{\mu _ n\} _ {n=1}^\infty of measures on (E, \mathcal E) converge weakly or does at least have a weak limit point? Evidently, a necessary condition is that \{\mu _ n(E)\} _ {n=1}^\infty is bounded. Hence, without loss of generality, we will consider only sequences in \mathcal M _ {\leqslant1}(E), as we did before. However, this condition is not sufficient for the existence of weak limit points, as for example the sequence \{\delta _ n\} _ {n\in\mathbb N} of probability measures on \mathbb R does not have a weak limit point (although it converges vaguely to the zero measure). This example suggests that we also have to make sure that no mass ''vanishes at infinity''. The idea will be made precise by the notion of tightness.

Recall that a family \mathcal F of measures is called weakly relatively sequentially compact if every sequence in \mathcal F has a weak limit point (in the closure of \mathcal F).

If E is compact, then \mathcal M _ 1(E) and \mathcal M _ {\leqslant1}(E) are tight, so they are weakly sequentially compact.

When E=\mathbb R, the following Helly's theorem is useful.

Recall that in numerical sequences, if a sequence of real numbers such that every subsequence that tends to a limit (\pm\infty allowed) as the same value for the limit; then the whole sequence tends to this limit. In particular a bounded sequence such that every convergent subsequence has the same limit is convergent to this limit. The next theorem generalizes this result to vague convergence of sub-probability measures in \mathbb R.

To prove the theorem by contraposition, suppose \mu _ n does not converge vaguely to \mu. Then it can be proved that there exists a continuity interval (a,b) of \mu such that \mu _ n(a,b) does not converge to \mu(a,b), so there exists a subsequence \{\mu _ {n _ k}\} such that it converges to a limit L different from \mu(a,b). From \{\mu _ {n _ k}\} there exists a sub-subsequence, say \{\mu _ {n _ {k(n')}}\}, which converges vaguely to \mu by hypothesis of the theorem. Hence \mu _ {n _ {k(n')}}(a,b)\to\mu(a,b). But the left side also converges to L, which is a contradiction.

We end this section with the following theorem on the connection between weak and vague convergence.

3. Relationships among various modes of convergence In section 5.1, we have some implications on convergence a.e. and convergence in probability. In this section we investigate more on the relationships among various modes of convergence. We first need one more notion of convergence of random variables: convergence in distribution.

Note that the defining relation in the above definition implies F(\infty)\leqslant\liminf F _ n(\infty). Hence if F _ n\to F in distribution, then F(\infty)=\lim F _ n(\infty).

For the connection between these two definitions, we have:

The continuous-mapping theorem is a simple result, but it is extremely useful. If the sequence of X _ n converges to X and g is continuous, then g(X _ n) converges to g(X). This is true for each of the three modes of stochastic convergence.

Proof. First note that U _ \varphi\subseteq E _ 1 is Borel measurable. Hence the conditions make sense.

Denote the probability measures of X,X _ 1,\dots by \mu,\mu _ 1,\dots. Then \mu(U _ \varphi)=0. Let f\in C _ b(E _ 2). Then f\circ \varphi is bounded and measurable, and its set of discontinuity must be ''smaller'' because every x continuous in \varphi is continuous in f\circ\varphi. Hence \mu(U _ {f\circ\varphi})=0. By Theorem 15,
\[
\lim _ {n\to\infty}\int f\, \mathrm d(\mu _ n\circ\varphi^{-1})=\lim _ {n\to\infty}\int(f\circ\varphi)\, \mathrm d\mu _ n=\int(f\circ\varphi)\, \mathrm d\mu=\int f\, \mathrm d(\mu\circ\varphi^{-1}).
\]
Therefore \mu _ n\circ\varphi^{-1}\stackrel{w}{\to}\mu\circ\varphi^{-1}, again by Theorem 15. (1) holds since \mu _ {\varphi(X)}=\mu _ X\circ\varphi^{-1}.

Now we consider (2). Fix \varepsilon>0, we want to show \mathbb P[d(\varphi(X _ n),\varphi(X))\geqslant\varepsilon]\to0 as n\to\infty. Suppose \delta>0. Consider the complement of the set \{x\in E _ 1\mid d(x,y)<\delta\Longrightarrow d(\varphi(x),\varphi(y))<\varepsilon\}, denoted by B _ \delta. If d(\varphi(X _ n),\varphi(X))\geqslant\varepsilon, then either d(X _ n,X)\geqslant\delta, or X\in B _ \delta. Consequently,
\[
\mathbb P[d(\varphi(X _ n),\varphi(X))\geqslant\varepsilon]\leqslant\mathbb P(X\in B _ \delta)+\mathbb P(d(X _ n,X)\geqslant\delta).
\]
By the definition of convergence in probability, \mathbb P(d(X _ n,X)\geqslant\delta)\to0 as n\to\infty. Then we let \delta\to0, and we obtain \mathbb P(X\in B _ \delta)\to0 as \delta\to0 because B _ \delta\cap U _ \varphi^c\downarrow\varnothing by the continuity of \varphi.

Assertion (3) is trivial.

Proof. (1) The first part: see Section 5.1; the second: d(X,X _ n)\stackrel{p}{\to}0 and X\stackrel d\to X. Then apply (3).

(2) We just need to show the ''if'' part. \mathbb P(|X _ n-c|\geqslant\varepsilon)=\mathbb P(X _ n\in B(c,\varepsilon)^c). By the (4) part of Theorem 15, \limsup \mathbb P(X _ n\in F)\leqslant\mathbb P(X\in F) for all closed set F. The completion of an open ball is clearly a closed set, so \limsup\mathbb P(X _ n\in B(c,\varepsilon)^c)\leqslant\mathbb P(c\in B(c,\varepsilon)^c)=0.

(3) We prove by showing that \mathbb E(f(X _ n)) and \mathbb E(f(Y _ n)) have the same limit for all bounded and Lipschitz continuous functions, and applying the (2) part of Theorem 15. For f with sup-norm M and Lipcshitz constant K, we take some \varepsilon>0, then
\begin{align*}
&|\mathbb E(f(X _ n))-\mathbb E(f(Y _ n))| \leqslant\mathbb E|f(Y _ n)-f(X _ n)| \\
={}& \mathbb E[|f(Y _ n)-f(X _ n)|\cdot\chi _ {\{|X _ n-Y _ n|<\varepsilon\}}]+\mathbb E[|f(Y _ n)-f(X _ n)|\cdot\chi _ {\{|X _ n-Y _ n|\geqslant\varepsilon\}}]\\
\leqslant{}& K\varepsilon\, \mathbb E\chi _ {\{|X _ n-Y _ n|<\varepsilon\}}+2M\, \mathbb P(d(X _ n,Y _ n)\geqslant\varepsilon)
\leqslant K\varepsilon+2M\, \mathbb P(d(X _ n,Y _ n)\geqslant\varepsilon).
\end{align*}
Let n\to\infty. The second term converges to zero. Then let \varepsilon\to0. The first term also converges to zero.

(4) Note that d[(X _ n,Y _ n),(X _ n,c)]=d(Y _ n,c)\stackrel{p}\to0. Thus, according to (3), it suffices to show that (X _ n, c)\stackrel{d}{\to}(X,c). For every bounded continuous function f(x,y), the function x\mapsto f(x,c) is bounded and continuous. If X _ n\stackrel d\to X, then \mathbb E[f(X _ n,c)]\to\mathbb E[f(X,c)]. This implies (X _ n, c)\stackrel{d}{\to}(X,c).

(5) This follows from d[(x _ 1,y _ 1),(x _ 2,y _ 2)]\leqslant d(x _ 1,x _ 2)+d(y _ 1,y _ 2).

Uniform integrability
We now introduce the concept of uniform integrability, which is of basic importance in some later theorems.

Here are some simple criteria. Finite family: If \mathcal F is a finite set, then it is uniformly integrable; Dominated: If \mathcal F is uniformly integrable and for any g\in\mathcal G there exists an f\in\mathcal F with |g|\leqslant|f|, then \mathcal G is also uniformly integrable.

Let p\in[1,\infty]. A family \mathcal F\subseteq L^p is called bounded in L^p if \sup\{|f| _ p\mid f\in\mathcal F\}<\infty. When p>1, if \mathcal F is bounded in L^p, then \mathcal F is uniformly integrable.

We are now ready for the following theorem.

Finally, we end this chapter with some results in convergence of moments. The function |x|^r\, (r>0) is continuous but not bounded, so we cannot apply the definition of weak convergence.