Directed information
Directed information,, is a measure of information theory that measures the amount of information that flows from the process to, where denotes the vector and denotes. The term directed information was coined by James Massey and is defined as
where is the conditional mutual information.
The Directed information has many applications in problems where causality plays an important role such as capacity of channel with feedback, capacity of discrete memoryless networks with feedback, gambling with causal side information, compression with causal side information,
and in real-time control communication settings , statistical physics.Estimation and Optimization of Directed Information
Estimating and optimizing the directed information is challenging because it is an expression of multi-letter, namely, it contains terms and as grows its more and more challenging. There exists algorithms for optimizing the directed information based on Blahut-Arimoto algorithm such as where the main idea is to start with the last element of the directed information and go backward.
For estimation there exists an algorithm based on context tree weight and on empirical
parametric distributions.
