# Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

721 questions
Filter by
Sorted by
Tagged with
17 views

11 views

### Mini batches and loss in recurrent neural networks (RNNs)

Suppose that we have a sequence $\left\{x^{(k)}\right\}_{k = 1}^{N}$ and that we wish to use a RNN to predict the next element of the sequence given the previous elements of the sequence (e.g., a ...
9 views

### Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
30 views

### Derivation in paper Deep Neural Networks as Gaussian processes in ICLR 2018

I am trying to understand the derivation of the main equation in the seminal paper titled Deep Neural Networks as Gaussian processes (in ICLR 2018). Following is the equation number (7), which can be ...
88 views

### How to grow the intuition behind this proof?

My intension is to understand this complete proof step by step in a lucid manner. I am trying few days to capture this, Unfortunatly I am failed. Can I hope to get a nice intuitive explanation of this ...
1 vote
28 views

### Can a neural network with ReLU activation represents exactly all $B$-bounded and $L$-Lipschitz $K$-max-affine functions?

A max-affine function is defined as the maximum over a set of affine functions, which is always convex. More specifically, we define a $K$-max-affine function $f:\mathbb{R}^d\to\mathbb{R}$ that can be ...
1 vote
13 views

### Universal approximation of neural networks

I am currently dealing with the topic of reproducing kernel Hilbert spaces (RKHS) given the draft book of Francis Bach. As a background knowledge for my current problem define: \begin{align} &H_1=\...
11 views

### Why overparametrization is necessary if one wants to interpolate the data smoothly?

For my research work, I am reading this paper named A Universal Law of Robustness via Isoperimetry. Solving n equations generically requires only n unknowns. But I got stuck in this line However, the ...
10 views

### What are the MATLAB syntax for various neural network learning algorithms and activation functions??

I just started learning MATLAB for neural network implementation. I just studied the various activation functions. However, I am confused in their syntax. Please help...
1 vote
15 views

### Derivation of the integral form of the numerator in the Bayesian inference equation???? (not on the denominator)

In the reference Gaussian processes: iterative sparse approximations by Csató, Lehel (Csató, Lehel. Gaussian processes: iterative sparse approximations. Diss. Aston University, 2002), on page 20, ...
14 views

### Confusion regarding the derivation of graph convolution

I am currently studying Spectral Graph Convolutions, and I am following this document: https://atcold.github.io/pytorch-Deep-Learning/en/week13/13-1/. They have derived the convolution as follows: The ...
53 views

### Is it possible (in principle and in meaningful way) to describe any subset of n-dimensional real Euclidean space?

Let us start with some background and motivation. My main question is very simple and it is available few paragraphs further and it is written in bold. My problem is based from the emerging theory of ...
39 views

### Understanding this Graph: What is a PetaFlop?

I was looking at this paper (https://arxiv.org/pdf/2005.14165.pdf) and came across this graph: I am trying to understand the following two things about this graph: What is PetaFLOP/s-days? I read ...
132 views

### Comparing the Training Costs of Machine Learning Algorithm: A Mathematical Perspective

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks. 1) Kernel Methods: For instance, I was looking at the optimization in ...
31 views

### Notation to indicate input and output dimension of a function

I have quite a few functions $F_i:\mathbb{R}^n\to\mathbb{R}^m$, where $n$ and $m$ aren't the same for each $F_i$. e.g. $F_0:\mathbb{R}^2\to\mathbb{R}^2$, $F_2:\mathbb{R}^3\to\mathbb{R}^4$, etc. I'm ...
21 views

### Isomorphic Neural Nets

The following is from Reconstructing a neural net from its output by Fefferman. In here, I'm not sure about the notation. Are we fixing one $l$ such that $\gamma_l$ is identity, and the rest of the ...
81 views

### Is "Probability Theory" an Inseparable Aspect of Machine Learning? [closed]

I have always had the following question about Probability and Machine Learning. As a simple example, suppose we have some data (e.g. heights of students: 175 cm, 181 cm, 162 cm, etc.) . If we assume ...
25 views

### How come nonlinear optimization problems need careful choices of initial parameters but neural networks appear to not have this issue?

When I run some nonlinear optimization code - I often encounter people saying that there is no global nonlinear optimization code that is guaranteed to reach a global maxima. Instead it is recommended ...
27 views

### Derivative of L1-loss

If y is the ground truth label vector and ŷ the predicted label vector. Then ...
14 views

### Backpropagation through time example (Recurrent Neural Network)

can some one help me understand & calculate the back propagation through time algorithm for this example (ever values are just scalars): The unit has an input $x_t$, a hidden state $h_{t-1}$, ...
90 views

### Can Non-Convex Optimization Problems have Closed Form Solutions?

In the realm of statistical modelling, creating a statistical model with respect to some data involves optimizing some mathematical function (e.g. Loss Function, OLS Equation, Maximum Likelihood ...
18 views

1 vote
62 views

### Vector-matrix differentiation and vectorisation

In recurrent neural network backpropagation (BPTT), we have the equations: \begin{align} e_t &= E^T x_t \\ a_t &= W_{hx}^T e_t+ W_{hh}^T h_{t-1}\\ h_t &= \text{tanh}(a_t) \\ s_t &= W_{...
1 vote
35 views

### Matrix Derivation for Neural Network Formula

I am learning some insights of Neural network but I have some problem with the derivation of matrix for backpropagation. On an assumption that the formula for calculating for one node in a neural ...
1 vote
41 views

### A proof involving Batch-Normalization and SGD in Neural Networks

I am trying to understand a proof from this paper. Consider the following setting: We train a neural network layer with SGD, that is by updating the weights according to w_{t+1} = w_{t} - \eta \...
1 vote
43 views

### Why to normalize an adjacency matrix?

In Kipf & Welling (2017) paper https://arxiv.org/pdf/1609.02907.pdf. It uses the normalized adjacency matrix $\mathbf{A}_{symm} = \mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2}$. I know the largest ...
1 vote
38 views

### Interpretation of an undirected adjacency matrix

I am new and know not much about "graph theory" and "graph neural network". Assume, I have one incidence matrix $\mathbf{B}$ such as visitor item1 item2 item3 item4 A 1 0 0 1 B ...
17 views

### Neural Networks in Mathematics

Neural networks are used extensively in machine learning and allied fields. I would like to know if neural networks crop up in an entirely non-machine learning setting. Such a setting should exclude ...
1 vote
20 views

### Derivative of a Vector Function (Hinge) [closed]

I need to calculate the derivative of the Hinge loss function, but the formulation is quite unfriendly. Here's how it's been defined for an instance where the $i$-th index corresponds to the right ...
43 views

### Why does my regression-NN completely fail to predict some points?

I would like to train a NN in order to approximate an unknown function $y = f(x_1,x_2)$. I have a lot of measurements $y = [y_1,\dots,y_K]$ (with K that could be in the range of 10-100 thousands) ...