Questions tagged [information-geometry]

Information theory is a study of probability and statistics using the techniques of differential geometry. The area covers statistical model, Fisher metric, $\alpha$ connection, canonical divergence etc.

Filter by
Sorted by
Tagged with
1
vote
0answers
30 views

Differentiating log likelihood with respect to data

I'm working with the probabilistic model derived from the KL NMF, in which we sample each entry of a data matrix X from the distribution $\forall_{i,j},X_{ij}\sim\text{Poisson}\left(\left[WH\right]_{...
2
votes
0answers
44 views

What is the intuition behind the Fisher-Rao metric?

I've seen the Fisher-Rao metric introduced via the following argument (see Sidhu and Kok 2019): There is a natural pairing between the simplex and its dual space of classical random variables: $\...
0
votes
0answers
31 views

A geometric property of divergence (statistics)

I am reading this wikipedia article about Divergence in statistics and I am a bit confused about the paragraph Geometrical Properties. In particular I can't understand the following $$ D[\partial_i \...
1
vote
1answer
40 views

Show the equivalence between two definitions of the Bures (/Fisher/Wasserstein) metric

I would like to understand the connection between these two, very different, definitions of the Bures distance. Through the traces of the density matrices $\rho_1$, $\rho_2$: \begin{equation} d(\...
1
vote
1answer
36 views

Is it true that $D_{KL}[q_1 \| p]\leq D_{KL}[q_2 \| p] \Rightarrow D_{KL}[p\|q_1]\leq D_{KL}[p \| q_2]$?

The KL divergence between two probability measures $p, q$ on a Polish space $E$ is defined as $$D_{KL}[q \| p] = \int_E \log \left(\frac{dq}{dp}\right)dq$$ provided that $q \ll p$. Now is it true that ...
2
votes
0answers
27 views

Reversed version of de Bruijn's identity/Boltzmann H-theorem for Markov processes

Take a Markov process $X_t$ on a Polish state-space $E$. Suppose the Markov process has invariant measure $\mu$ and is prepared with initial distribution $\mu_0\ll \mu$. De Bruijn's identity says that ...
0
votes
0answers
27 views

KL Divergence estimation

I am trying to construct a certain bound for the KL divergence between two two numbers. I want to show that $-n D((a+s/\sqrt{n}|| a)\le -\frac{s}{2a(1-a)}+Error(1/\sqrt{n})$ $(a\in(0,1))$ \begin{align*...
1
vote
0answers
67 views

Stationary distributions of a Markov chain are $e-$convex?

Consider a Markov chain with discrete time and discrete state space, with transition matrix $M$. A probability distribution on states $\pi$ is said to be an invariant distribution when $\pi M=\pi$. It ...
4
votes
1answer
153 views

What do the level sets of the Shannon entropy look like?

The Shannon entropy of a discrete probability distribution $\newcommand{\bs}[1]{\boldsymbol{#1}}\bs p\equiv (p_i)_{i=1}^n$ is defined as $H(\bs p)\equiv -\sum_{i=1}^n p_i \log p_i$. Consider the ...
1
vote
0answers
38 views

How does curvature affect KL Divergence?

I have recently started studying Information Geometry and I came across this blog Natural Gradient Descent which talks about the Hessian acting as the curvature of KL divergence. I was wondering, is ...
3
votes
1answer
43 views

Counterexample to finiteness of KL divergence?

Suppose $P,Q$ are two probability distributions on a measurable space $(X, \mathcal F)$. The KL-divergence between $P$ and $Q$ is defined as \begin{equation} D_{KL}[P||Q] = \int_X \log \frac {dP}{dQ}...
6
votes
1answer
290 views

Examples of non-positively Curvature Riemannian Manifolds

When I read about complete, simply connected, and connected Riemannian manifolds of non-positive curvature I only find explicit examples of hyperbolic $n$-space and Euclidean space. What are other ...
2
votes
1answer
113 views

Amari's Pythagorean theorem

Recently I started to read Amari's "Information Geometry and Its Applications". I quickly stumbled upon some (apparent) inconsistencies. It starts with the "generalized Pythagorean theorem" on a ...
1
vote
1answer
82 views

Calculating the tangent space of the manifold of probability measures on a finite set

Let $I$ be a finite set, $\mathcal{F}(I):= \{I\to\mathcal{R}\}$ the vector space of functions on $I$ with basis $e_i$, where $e_i(i)=1, e_i(j) = 0 (i \neq j)$. Let $\mathcal{S}(I)$ be the ...
1
vote
1answer
75 views

Why must a manifold be orientable to be able to induce a specific statistical manifold?

Let $M$ be a manifold, $g$ a Riemannian metric, $\nabla$ be an affine connection on $M$, and $\nabla^{*}$ the unique dual affine connection of $\nabla$ on $M$, i.e. for all vector fields $X,Y,Z$ on $M$...
1
vote
1answer
53 views

Uniqueness condition for the em algorithm in information geometry

In his book, Information Geometry and its Applications, p. 28, Amari makes a statement without proof (quoted in the form of an image below) that seems to be incorrect. I would like to know whether the ...
1
vote
1answer
116 views

Relationship between symmetrized KL-divergence and geodesics

I am currently working through Amari's Information Geometry and its Applications and in chapter 3, theorem 3.2 states that for distributions $p, q$ on discrete symbols, the following relationship ...
1
vote
0answers
22 views

Projection of a probability measure space onto its subspace via Wasserstein metric has a unique solution?

Let $(\mathcal{X}, d)$ be a Polish space, $p \geq 1$, $\mathcal{P}_p(\mathcal{X})$ be the space of probability measures over $\mathcal{X}$ such that the $p$-Wasserstein metric between any two ...
3
votes
1answer
87 views

Is an exponential family the same as an exponentially convex set?

An exponential family of probability distributions is usually defined as $$ p(x) = e^{\theta\cdot f(x) \,-\, \psi(\theta)}, $$ where $\theta$ is a vector of parameters, $f(x)$ is some arbitrary vector-...
1
vote
0answers
42 views

Is the Wasserstein distance the same down a Markov chain?

For two Markov chains: $$X \rightarrow f(X)$$ $$Y \rightarrow f(Y)$$ where $f$ is 1-Lipschitz, if one minimizes the Wasserstein distance between $f(X)$ and $f(Y)$, does this result in the ...
2
votes
0answers
88 views

Intuition behind defining the information of a random variable as being additive for independent events

I'm recently studying the concepts of entropy and I've a fundamental question regarding the conceptual formulation of information content of a random variable $X$, or equivalently, the uncertaintly of ...
7
votes
1answer
487 views

Relation between information geometry and geometric deep learning

I'm currently working on information geometry (IG) and geometric deep learning (GDL). As I started without specific knowledge of both, their respective names led me to believe for a short and naive ...
1
vote
1answer
182 views

Bounds on KL divergence (behavior like a distance)

The KL divergence is not a distance metric due to the fact that it violates the triangle inequality and since it is not symmetric. However, consider the following problem. Let $A$ and $B$ be two ...
3
votes
2answers
209 views

Variational inference: Does the natural gradient follow geodesics locally?

Amari's natural gradient descent is a well-known optimisation algorithm from information geometry that is well-suited for finding optima of functionals on statistical manifolds. It consists of ...
3
votes
1answer
57 views

Explanation of one the criteria for a function to be a divergence on a manifold

Reading Definition 1.1 in Information theory and its applications (2016), by Amari, Shun-ichi, the following is the definition of a divergence, that is, an asymmetric measure of distance $D$ on a ...
4
votes
1answer
257 views

Statistical distance induced by Fisher information metric on statistical manifold of categorical distribution (simplex)

I am trying to compute the information length or distance induced by the Fisher information metric on the statistical manifold of the categorical distribution (the interior of the n-dimensional ...
0
votes
2answers
691 views

upper bound on cross entropy or relative entropy

Am looking for upper bounds for relative-entropy or cross-entropy for non-gaussian case . All I am aware of is bounds for entropy based on determinant of covariance matrix. What about for relative ...
2
votes
1answer
194 views

Simple question on parallel transport in dually flat manifolds

I just started studying Information Geometry and its applications by Amari. Right in the first chapter, the author talks about parallel transport in Dually flat manifolds. Just some quick notation: ...
0
votes
1answer
49 views

A basic question on mutually orthogonal coordinate systems

I am reading the first chapter of Information Geometry and its applications by Amari. I am struggling to grasp a basic concept about mutually orthogonal coordinate systems. Since the book is not ...
0
votes
1answer
140 views

Exponential map on the Fisher manifold for exponential family distribution

So, I don't really understand too well Diff. Geometry and Manifolds currently. Hence, I've started studying it and it is very interesting. However, atm I just need to understand how to compute the ...
2
votes
1answer
242 views

Is the quantum relative entropy a Bregman divergence?

This is related to a broader question about quantum information geometry - this question is more specific. A fundamental concept in Amari's treatment of information geometry is that of a Bregman ...
17
votes
1answer
478 views

What is the essential difference between classical and quantum information geometry?

This question may be a little subjective, but I would like to understand, from a geometric perspective, how the structure of quantum theory differs from that of classical probability theory. I have a ...
3
votes
1answer
302 views

Connections and applications of/between information (field) theory and information geometry

I hope this isn't any kind of duplicate. I have looked through the forum, but it's possible I overlooked something. I'm currently in the 4th semester of my physics studies and listened to some ...
2
votes
2answers
218 views

Flatness of a statistical manifold with Fisher information metric

Let $\mathcal{M} = \{p_\theta := p(\cdot | \theta), \theta \in \Theta\}$ be a statistical manifold with Fisher information metric: $$g_{{jk}}(\theta )=\operatorname {E} \left[\left({\frac {\partial }{\...
1
vote
0answers
78 views

$\alpha$ pythagorean theorem for $\alpha$ divergence in the probability simplex $S_n$

I'm trying to understand "Methods of information geometry" of Amari and Nagaoka, p72. Consider the probability simplex $$S_n:=\{p=(p_1,\cdots,p_{n+1})\in \mathbb{R}^{n+1}~|~\sum_i^{n+1} p_i=1,~p_i\...
0
votes
1answer
63 views

Find parameter transformation of probability distribution such that the transformed parameters are orthogonal

I'm looking for a parameter transformation of a probability distribution such that the resulting parameters are orthogonal. That is, the off-diagonal elements of the Fisher Information matrix of the ...
2
votes
1answer
23 views

local geometry interpretation of binary hypothesis testing

In information geometry, suppose $$ \mathcal{N}_{\epsilon}^{\mathcal{X}}\triangleq\left\{P\in \mathcal{P}^{\mathcal{X}}:\sum_{x\in\mathcal{X}}\frac{(P(x)-P_0(x)^2}{P_0(x)}\leq \epsilon^2\right\} $$ ...
2
votes
0answers
123 views

Maximizing a mutual information w.r.t. (i.i.d.) variation of the channel.

Intuitively, I have I have a discrete (finite) random variable $Z$ whose probability mass function $q(z)$ I can choose. $Z$ selects the conditional probability distribution $p(y|x,z)$ of another (...
1
vote
1answer
107 views

Do the set of all factorisable distributions form an exponential family?

A (natural) exponential family of probability distributions is defined as $$ p(x|\theta) = \exp\big(g(x) + \theta\cdot f(x) - \psi(\theta)\big),\tag{1} $$ where $\theta$ is a vector of parameters and $...
3
votes
0answers
149 views

What is the connection between information monotonicity and the additivity property of information measures?

First of all, sorry if the question is trivial or not clear. I tried to be as clear as possible, but I'm a beginner in information geometry, with a background in physics, self-studying information ...
2
votes
0answers
155 views

Coordinate independent definition of Fisher metric on statistical manifolds

Is there a manifestly coordinate independent definition of the Fisher metric? I was reading Methods of Information Geometry by Amari and Nagaoka and Information Geometry and Its Applications by Amari, ...
3
votes
1answer
196 views

Basis vectors in information geometry

A basis vector in the tangent space at a point of a smooth manifold is given by a differential operator such as $\partial_{i}=\cfrac{\partial }{\partial \xi^{i}}$. On the other hand, in a statistical ...
4
votes
1answer
703 views

Pythagoras theorem holds with equality for linear families

The Pythagoras theorem for information projections is given by: $$D(p||q) \geq D(p||p^*) + D(p^*||q)\;\; \forall p,$$ where $p^* = argmin_{r \in {\cal S}} D(r||q)$, or the projection of the ...
4
votes
1answer
186 views

Wasserstein-like measure for disconnected spaces

Does there exist a Wasserstein-like measure for difference between probability distributions, where the space on which the probabilities are defined is not connected? E.g. a distribution over six ...
-1
votes
1answer
66 views

the output dependency of DMC channel when the input components are independent with each other.

Most textbooks don't assert that the distributions of components of the output sequence $Y_{1},Y_{2},..Y_{n}$ of the discrete memoryless channel(DMC) are independent with each other even if the ...
2
votes
0answers
179 views

Monotonicity of the Jensen-Shannon divergence

Let $\text{JSD}(P\mid\mid Q)$ be the Jensen-Shannon divergence (https://en.wikipedia.org/wiki/Jensen-Shannon_divergence) between two probability distributions $P$ and $Q$. Question: Is $\text{JSD}(P\...
7
votes
1answer
381 views

How many types of mathematical information are there?

I am aware of at least two types of mathematical information: Shannon Information, which is the negative of entropy (i.e. a loss of entropy by $n$ bits is precisely a gain of Shannon information of $...
2
votes
2answers
173 views

Question on the motivation of Information Geometry

In the Preface of 'Methods of Information Geometry': We consider the set, $S$, of normal distributions $P(x; \mu, \sigma) = \dfrac{1}{\sqrt{2\pi }\sigma}\exp \left \{ - \dfrac{(x-\mu)^2}{2\sigma^2}\...
12
votes
1answer
1k views

Self-studying Information Geometry

I was recently exposed to the topic of Information Geometry by a friend of mine, and was looking for a good book to begin self-studying this topic. Any suggestions? Also, what subject matter would ...
12
votes
1answer
881 views

Higher math and statistics/probability

So I've heard that certain areas of statistics and probability use manifolds and results from analysis and topology. Given that I lack the background to see where manifolds would become useful in ...