Chernoff bound


In probability theory, the Chernoff bound, named after Herman Chernoff but due to Herman Rubin, gives exponentially decreasing bounds on tail distributions of sums of independent random variables. It is a sharper bound than the known first- or second-moment-based tail bounds such as Markov's inequality or Chebyshev's inequality, which only yield power-law bounds on tail decay. However, the Chernoff bound requires that the variates be independent – a condition that neither Markov's inequality nor Chebyshev's inequality require, although Chebyshev's inequality does require the variates to be pairwise independent.
It is related to the Bernstein inequalities and to Hoeffding's inequality.

The generic bound

The generic Chernoff bound for a random variable is attained by applying Markov's inequality to. For every :
When is the sum of random variables, we get for any t > 0,
In particular, optimizing over t and assuming that are independent, we obtain,
Similarly,
and so,
Specific Chernoff bounds are attained by calculating for specific instances of the basic variables.

Example

Let be independent Bernoulli random variables, whose sum is, each having probability p > 1/2 of being equal to 1. For a Bernoulli variable:
So:
For any, taking and gives:
and the generic Chernoff bound gives:
The probability of simultaneous occurrence of more than n/2 of the events has an exact value:
A lower bound on this probability can be calculated based on Chernoff's inequality:
Indeed, noticing that, we get by the multiplicative form of Chernoff bound,
This result admits various generalizations as outlined below. One can encounter many flavors of Chernoff bounds: the original additive form or the more practical multiplicative form.

Additive form (absolute error)

The following Theorem is due to Wassily Hoeffding and hence is called the Chernoff–Hoeffding theorem.
A simpler bound follows by relaxing the theorem using, which follows from the convexity of and the fact that
This result is a special case of Hoeffding's inequality. Sometimes, the bounds
which are stronger for are also used.

Multiplicative form (relative error)

A similar proof strategy can be used to show that
The above formula is often unwieldy in practice, so the following looser but more convenient bounds are often used:
which follow from the inequality from the list of logarithmic inequalities.
Or looser still:

Applications

Chernoff bounds have very useful applications in set balancing and packet routing in sparse networks.
The set balancing problem arises while designing statistical experiments. Typically while designing a statistical experiment, given the features of each participant in the experiment, we need to know how to divide the participants into 2 disjoint groups such that each feature is roughly as balanced as possible between the two groups. Refer to this for more info on the problem.
Chernoff bounds are also used to obtain tight bounds for permutation routing problems which reduce network congestion while routing packets in sparse networks. Refer to this for a thorough treatment of the problem.
Chernoff bounds are used in computational learning theory to prove that a learning algorithm is probably approximately correct, i.e. with high probability the algorithm has small error on a sufficiently large training data set.
Chernoff bounds can be effectively used to evaluate the "robustness level" of an application/algorithm by exploring its perturbation space with randomization.
The use of the Chernoff bound permits one to abandon the strong -and mostly unrealistic- small perturbation hypothesis. The robustness level can be, in turn, used either to validate or reject a specific algorithmic choice, a hardware implementation or the appropriateness of a solution whose structural parameters are affected by uncertainties.

Matrix bound

and Andreas Winter introduced a Chernoff bound for matrix-valued random variables. The following version of the inequality can be found in the work of Tropp.
Let be independent matrix valued random variables such that and.
Let us denote by the operator norm of the matrix. If holds almost surely for all, then for every
Notice that in order to conclude that the deviation from 0 is bounded by with high probability, we need to choose a number of samples proportional to the logarithm of. In general, unfortunately, a dependence on is inevitable: take for example a diagonal random sign matrix of dimension. The operator norm of the sum of t independent samples is precisely the maximum deviation among d independent random walks of length t. In order to achieve a fixed bound on the maximum deviation with constant probability, it is easy to see that t should grow logarithmically with d in this scenario.
The following theorem can be obtained by assuming M has low rank, in order to avoid the dependency on the dimensions.

Theorem without the dependency on the dimensions

Let and M be a random symmetric real matrix with and almost surely. Assume that each element on the support of M has at most rank r. Set
If holds almost surely, then
where are i.i.d. copies of M.

Theorem with matrices that are not completely random

Garg, Lee, Song and Srivastava proved a Chernoff-type bound for sums of matrix-valued random variables sampled via a random walk on an expander, confirming a conjecture due to Wigderson and Xiao.
Kyng and Song proved a Chernoff-type bound for sums of Laplacian matrix of random spanning trees.

Sampling variant

The following variant of Chernoff's bound can be used to bound the probability that a majority in a population will become a minority in a sample, or vice versa.
Suppose there is a general population A and a sub-population B⊆A. Mark the relative size of the sub-population by r.
Suppose we pick an integer k and a random sample S⊂A of size k. Mark the relative size of the sub-population in the sample by rS.
Then, for every fraction d∈:
In particular, if B is a majority in A we can bound the probability that B will remain majority in S by taking: d = 1 - 1 / :
This bound is of course not tight at all. For example, when r=0.5 we get a trivial bound Prob > 0.

Proofs

Chernoff–Hoeffding theorem (additive form)

Let. Taking in, we obtain:
Now, knowing that, we have
Therefore, we can easily compute the infimum, using calculus:
Setting the equation to zero and solving, we have
so that
Thus,
As, we see that, so our bound is satisfied on. Having solved for, we can plug back into the equations above to find that
We now have our desired result, that
To complete the proof for the symmetric case, we simply define the random variable, apply the same proof, and plug it into our bound.

Multiplicative form

Set.
According to,
The third line above follows because takes the value with probability and the value 1 with probability. This is identical to the calculation above in the proof of the Theorem for additive form.
Rewriting as and recalling that , we set. The same result can be obtained by directly replacing in the equation for the Chernoff bound with.
Thus,
If we simply set so that for, we can substitute and find
This proves the result desired.