I'm currently working on information geometry (IG) and geometric deep learning (GDL). As I started without specific knowledge of both, their respective names led me to believe for a short and naive period that GDL was defined by the use of IG notions in deep learning. This now appears to me as substantially inaccurate, but since there are indeed various connections, I would like to clarify the relation between them.
In Geometric deep learning: going beyond Euclidean data, GDL is defined as:
Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds.
The article later defines Riemannian manifolds and metrics, calculus on manifolds, etc to complete the toolkit needed to build a machine learning favorable environment.
On the other hand, in Amari's Information Geometry and Its Applications, IG is described in the following paragraph:
Information geometry has emerged from studies of invariant geometrical structure involved in statistical inference. It defines a Riemannian metric together with dually coupled affine connections in a manifold of probability distributions. These structures play important roles not only in statistical inference but also in wider areas of information sciences, such as machine learning, signal processing, optimization, and even neuroscience, not to mention mathematics and physics.
In the same book, Amari mentions a neural manifold, containing all neural networks:
The set of all such networks forms a manifold, where matrix $W =w_{ji}$ is a coordinate system.
The underlying question is: to what extent is GDL related to IG ?
If I'm not mistaken, working on a riemannian manifold with a neural network, and relying on a riemannian gradient for its training, implies one is doing geometric deep learning, but not necessarily information geometry. An example of such GDL/non-IG neural network is SPD Net, which relies on SPD matrices for intermediate representations, defines new transformations aiming at keeping representations on a manifold (hence a bilinear mapping layer $W.X.W^T$ with $W$ belonging to a compact Stiefel manifold - cf. BiMap Layer section of the previous article), and depends on a riemannian gradient. It seems similar to IG since we speak of riemannian manifold and riemannian gradient, but it doesn't match the manifold of probability distributions aspect.
My current understanding is that GDL and IG are closely related since they happen to rely on similar mathematical objects, and perhaps more importantly, that in both cases we try to reach the right representation of a "latent subspace". The learning mechanisms of GDL then owe their success to optimization on the well chosen manifolds using gradients possibly defined in IG, the manifolds ideally describing a subspace where all possible data is living, not only the samples available for training. However, this relation is limited to specific cases of GDL, where a neural network uses manifolds and metrics also used in IG. One can notice that this completely excludes, among others, the graph-based part of GDL (?).
Finally, we could still find a direct relation between GDL and IG if we used neural networks to work on manifolds of probability distributions, which are the very objects that IG targets. That seems rather relevant for generative models, but it's a very specific case of GDL for which the high usefulness of IG is contextual.
Sorry in advance if this is a duplicate, I only checked the information-geometry tagged questions before posting mine. Since IG isn't that new anymore, and this is a rather basic definition question, I chose not to post it on mathoverflow.
Related AI SE post: What is geometric deep learning?, my answer there could be inaccurate depending on what the community will answer here.
Mathoverflow cross-post: here, note that the latter includes a disclaimer explaining why I allowed myself to cross-post.
