Questions tagged [information-geometry]

Information theory is a study of probability and statistics using the techniques of differential geometry. The area covers statistical model, Fisher metric, $\alpha$ connection, canonical divergence etc.

Filter by
Sorted by
Tagged with
4
votes
1answer
101 views

What do the level sets of the Shannon entropy look like?

The Shannon entropy of a discrete probability distribution $\newcommand{\bs}[1]{\boldsymbol{#1}}\bs p\equiv (p_i)_{i=1}^n$ is defined as $H(\bs p)\equiv -\sum_{i=1}^n p_i \log p_i$. Consider the ...
1
vote
0answers
26 views

How does curvature affect KL Divergence?

I have recently started studying Information Geometry and I came across this blog Natural Gradient Descent which talks about the Hessian acting as the curvature of KL divergence. I was wondering, is ...
3
votes
1answer
30 views

Counterexample to finiteness of KL divergence?

Suppose $P,Q$ are two probability distributions on a measurable space $(X, \mathcal F)$. The KL-divergence between $P$ and $Q$ is defined as \begin{equation} D_{KL}[P||Q] = \int_X \log \frac {dP}{dQ}...
0
votes
0answers
10 views

Bound $TV((\nabla \phi_1)_{\#}\mu_1,(\nabla \phi_2)_{\#}\mu_2)$ where $\phi_k: R^d \to R^d$ is $C^2$ with $\|Id - \nabla \phi_k\|_\infty \le u$

Working on a problem in machine-learning theory, I'm led to the following question. Let $\varepsilon \ge 0$ be given and for $k=1,2$, let $\phi_k:\mathbb R^d \to \mathbb R$ be functions such that $\...
6
votes
1answer
137 views

Examples of non-positively Curvature Riemannian Manifolds

When I read about complete, simply connected, and connected Riemannian manifolds of non-positive curvature I only find explicit examples of hyperbolic $n$-space and Euclidean space. What are other ...
0
votes
0answers
36 views

Can Kullback-Leibler distance comply triangle inequality when symmetrized with symmetric function?

The relative entropy between two probability mass functions $p(x)$ and $q(x)$ can be computed with the Kullback-Leibler distante $$ D(p||q) = \sum_x p(x) \log \frac{p(x)}{q(x)} $$ The KL distance ...
2
votes
1answer
56 views

Amari's Pythagorean theorem

Recently I started to read Amari's "Information Geometry and Its Applications". I quickly stumbled upon some (apparent) inconsistencies. It starts with the "generalized Pythagorean theorem" on a ...
1
vote
1answer
32 views

Calculating the tangent space of the manifold of probability measures on a finite set

Let $I$ be a finite set, $\mathcal{F}(I):= \{I\to\mathcal{R}\}$ the vector space of functions on $I$ with basis $e_i$, where $e_i(i)=1, e_i(j) = 0 (i \neq j)$. Let $\mathcal{S}(I)$ be the ...
1
vote
1answer
53 views

Why must a manifold be orientable to be able to induce a specific statistical manifold?

Let $M$ be a manifold, $g$ a Riemannian metric, $\nabla$ be an affine connection on $M$, and $\nabla^{*}$ the unique dual affine connection of $\nabla$ on $M$, i.e. for all vector fields $X,Y,Z$ on $M$...
0
votes
0answers
16 views

Are Iso-Cross-Entropy surfaces always a convex hull?

Are iso-cross-entropy surfaces always a convex hull? In other words, if I have a function that describes the points where cross-entropy is constant, is that function always convex? I suspect the ...
1
vote
1answer
37 views

Uniqueness condition for the em algorithm in information geometry

In his book, Information Geometry and its Applications, p. 28, Amari makes a statement without proof (quoted in the form of an image below) that seems to be incorrect. I would like to know whether the ...
1
vote
1answer
64 views

Relationship between symmetrized KL-divergence and geodesics

I am currently working through Amari's Information Geometry and its Applications and in chapter 3, theorem 3.2 states that for distributions $p, q$ on discrete symbols, the following relationship ...
1
vote
0answers
12 views

Projection of a probability measure space onto its subspace via Wasserstein metric has a unique solution?

Let $(\mathcal{X}, d)$ be a Polish space, $p \geq 1$, $\mathcal{P}_p(\mathcal{X})$ be the space of probability measures over $\mathcal{X}$ such that the $p$-Wasserstein metric between any two ...
3
votes
1answer
65 views

Is an exponential family the same as an exponentially convex set?

An exponential family of probability distributions is usually defined as $$ p(x) = e^{\theta\cdot f(x) \,-\, \psi(\theta)}, $$ where $\theta$ is a vector of parameters, $f(x)$ is some arbitrary vector-...
1
vote
0answers
31 views

Is the Wasserstein distance the same down a Markov chain?

For two Markov chains: $$X \rightarrow f(X)$$ $$Y \rightarrow f(Y)$$ where $f$ is 1-Lipschitz, if one minimizes the Wasserstein distance between $f(X)$ and $f(Y)$, does this result in the ...
2
votes
0answers
60 views

Intuition behind defining the information of a random variable as being additive for independent events

I'm recently studying the concepts of entropy and I've a fundamental question regarding the conceptual formulation of information content of a random variable $X$, or equivalently, the uncertaintly of ...
0
votes
1answer
37 views

What is the relationship between the information filter (inverse covariance filter) and the fisher information?

In the information filter (or inverse covariance filter), the information update is given by $$ \mathbf{Y}_k = \mathbf{Y}_{k-1} + \mathbf{I}_k $$ where $\mathbf{Y}_k$ is the inverse of the ...
5
votes
1answer
375 views

Relation between information geometry and geometric deep learning

I'm currently working on information geometry (IG) and geometric deep learning (GDL). As I started without specific knowledge of both, their respective names led me to believe for a short and naive ...
0
votes
1answer
102 views

Bounds on KL divergence (behavior like a distance)

The KL divergence is not a distance metric due to the fact that it violates the triangle inequality and since it is not symmetric. However, consider the following problem. Let $A$ and $B$ be two ...
3
votes
2answers
102 views

Variational inference: Does the natural gradient follow geodesics locally?

Amari's natural gradient descent is a well-known optimisation algorithm from information geometry that is well-suited for finding optima of functionals on statistical manifolds. It consists of ...
3
votes
1answer
49 views

Explanation of one the criteria for a function to be a divergence on a manifold

Reading Definition 1.1 in Information theory and its applications (2016), by Amari, Shun-ichi, the following is the definition of a divergence, that is, an asymmetric measure of distance $D$ on a ...
3
votes
1answer
150 views

Statistical distance induced by Fisher information metric on statistical manifold of categorical distribution (simplex)

I am trying to compute the information length or distance induced by the Fisher information metric on the statistical manifold of the categorical distribution (the interior of the n-dimensional ...
0
votes
1answer
338 views

upper bound on cross entropy or relative entropy

Am looking for upper bounds for relative-entropy or cross-entropy for non-gaussian case . All I am aware of is bounds for entropy based on determinant of covariance matrix. What about for relative ...
2
votes
1answer
105 views

Simple question on parallel transport in dually flat manifolds

I just started studying Information Geometry and its applications by Amari. Right in the first chapter, the author talks about parallel transport in Dually flat manifolds. Just some quick notation: ...
0
votes
1answer
44 views

A basic question on mutually orthogonal coordinate systems

I am reading the first chapter of Information Geometry and its applications by Amari. I am struggling to grasp a basic concept about mutually orthogonal coordinate systems. Since the book is not ...
1
vote
1answer
102 views

Exponential map on the Fisher manifold for exponential family distribution

So, I don't really understand too well Diff. Geometry and Manifolds currently. Hence, I've started studying it and it is very interesting. However, atm I just need to understand how to compute the ...
1
vote
1answer
188 views

Is the quantum relative entropy a Bregman divergence?

This is related to a broader question about quantum information geometry - this question is more specific. A fundamental concept in Amari's treatment of information geometry is that of a Bregman ...
15
votes
0answers
371 views

What is the essential difference between classical and quantum information geometry?

This question may be a little subjective, but I would like to understand, from a geometric perspective, how the structure of quantum theory differs from that of classical probability theory. I have a ...
3
votes
1answer
233 views

Connections and applications of/between information (field) theory and information geometry

I hope this isn't any kind of duplicate. I have looked through the forum, but it's possible I overlooked something. I'm currently in the 4th semester of my physics studies and listened to some ...
1
vote
2answers
161 views

Flatness of a statistical manifold with Fisher information metric

Let $\mathcal{M} = \{p_\theta := p(\cdot | \theta), \theta \in \Theta\}$ be a statistical manifold with Fisher information metric: $$g_{{jk}}(\theta )=\operatorname {E} \left[\left({\frac {\partial }{\...
1
vote
0answers
70 views

$\alpha$ pythagorean theorem for $\alpha$ divergence in the probability simplex $S_n$

I'm trying to understand "Methods of information geometry" of Amari and Nagaoka, p72. Consider the probability simplex $$S_n:=\{p=(p_1,\cdots,p_{n+1})\in \mathbb{R}^{n+1}~|~\sum_i^{n+1} p_i=1,~p_i\...
0
votes
1answer
55 views

Find parameter transformation of probability distribution such that the transformed parameters are orthogonal

I'm looking for a parameter transformation of a probability distribution such that the resulting parameters are orthogonal. That is, the off-diagonal elements of the Fisher Information matrix of the ...
2
votes
1answer
19 views

local geometry interpretation of binary hypothesis testing

In information geometry, suppose $$ \mathcal{N}_{\epsilon}^{\mathcal{X}}\triangleq\left\{P\in \mathcal{P}^{\mathcal{X}}:\sum_{x\in\mathcal{X}}\frac{(P(x)-P_0(x)^2}{P_0(x)}\leq \epsilon^2\right\} $$ ...
2
votes
0answers
107 views

Maximizing a mutual information w.r.t. (i.i.d.) variation of the channel.

Intuitively, I have I have a discrete (finite) random variable $Z$ whose probability mass function $q(z)$ I can choose. $Z$ selects the conditional probability distribution $p(y|x,z)$ of another (...
1
vote
1answer
73 views

Do the set of all factorisable distributions form an exponential family?

A (natural) exponential family of probability distributions is defined as $$ p(x|\theta) = \exp\big(g(x) + \theta\cdot f(x) - \psi(\theta)\big),\tag{1} $$ where $\theta$ is a vector of parameters and $...
2
votes
0answers
138 views

What is the connection between information monotonicity and the additivity property of information measures?

First of all, sorry if the question is trivial or not clear. I tried to be as clear as possible, but I'm a beginner in information geometry, with a background in physics, self-studying information ...
2
votes
0answers
145 views

Coordinate independent definition of Fisher metric on statistical manifolds

Is there a manifestly coordinate independent definition of the Fisher metric? I was reading Methods of Information Geometry by Amari and Nagaoka and Information Geometry and Its Applications by Amari, ...
3
votes
1answer
177 views

Basis vectors in information geometry

A basis vector in the tangent space at a point of a smooth manifold is given by a differential operator such as $\partial_{i}=\cfrac{\partial }{\partial \xi^{i}}$. On the other hand, in a statistical ...
3
votes
1answer
544 views

Pythagoras theorem holds with equality for linear families

The Pythagoras theorem for information projections is given by: $$D(p||q) \geq D(p||p^*) + D(p^*||q)\;\; \forall p,$$ where $p^* = argmin_{r \in {\cal S}} D(r||q)$, or the projection of the ...
3
votes
1answer
164 views

Wasserstein-like measure for disconnected spaces

Does there exist a Wasserstein-like measure for difference between probability distributions, where the space on which the probabilities are defined is not connected? E.g. a distribution over six ...
-1
votes
1answer
58 views

the output dependency of DMC channel when the input components are independent with each other.

Most textbooks don't assert that the distributions of components of the output sequence $Y_{1},Y_{2},..Y_{n}$ of the discrete memoryless channel(DMC) are independent with each other even if the ...
1
vote
0answers
165 views

Monotonicity of the Jensen-Shannon divergence

Let $\text{JSD}(P\mid\mid Q)$ be the Jensen-Shannon divergence (https://en.wikipedia.org/wiki/Jensen-Shannon_divergence) between two probability distributions $P$ and $Q$. Question: Is $\text{JSD}(P\...
7
votes
1answer
369 views

How many types of mathematical information are there?

I am aware of at least two types of mathematical information: Shannon Information, which is the negative of entropy (i.e. a loss of entropy by $n$ bits is precisely a gain of Shannon information of $...
2
votes
2answers
154 views

Question on the motivation of Information Geometry

In the Preface of 'Methods of Information Geometry': We consider the set, $S$, of normal distributions $P(x; \mu, \sigma) = \dfrac{1}{\sqrt{2\pi }\sigma}\exp \left \{ - \dfrac{(x-\mu)^2}{2\sigma^2}\...
11
votes
1answer
1k views

Self-studying Information Geometry

I was recently exposed to the topic of Information Geometry by a friend of mine, and was looking for a good book to begin self-studying this topic. Any suggestions? Also, what subject matter would ...
11
votes
1answer
712 views

Higher math and statistics/probability

So I've heard that certain areas of statistics and probability use manifolds and results from analysis and topology. Given that I lack the background to see where manifolds would become useful in ...
9
votes
2answers
825 views

reference request for a book on high dimensional probability and data analysis written for mathematicians

I hope someone can help with this. I am a statistician looking for a good book on high dimensional probability and data analysis. Basically I am looking for the equivalent of Terry Tao's 2 volume set ...
3
votes
1answer
464 views

Limit of the multinomial distribution on the simplex

Let $\Sigma_k$ be the $k$-dimensional simplex $\{x_1,\dots x_{k+1}| \sum_j x_1=1\}$. Given a set of parameters $\vec{q}=(q_1, \dots q_{k})$ in $\Sigma_{k-1}$ and a bunch of non-negative integers $\vec{...
0
votes
1answer
87 views

What modification is this of the notion of Renyi divergence?

Given two probability distributions $P$ and $Q$ over the same outcome and event space (assume finite if needed) one defines their Renyi divergence as $D_\alpha (P \vert \vert Q) = \frac{1}{\alpha -1} \...
21
votes
1answer
768 views

Manifolds with volume forms on every submanifold

If we equip a manifold with an inner product (i.e. we have a Riemannian Manifold) then we get a canonical volume form on that manifold (please mentally insert the prefix "pseudo" into my question ...