Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

How to get the Lipschitz constant $L$ using the inequality?

Taken from paper "A Universal Law of Robustness via isoperimetry" by Bubeck and Sellke. $$ \begin{aligned} &\mathbb{P}\left(\exists f \in \mathcal{F}: \frac{1}{n} \sum_{i=1}^{n}\left(y_{...
user avatar
1 vote
0 answers
63 views

How the second inequality stands?

Taken from paper "A Universal Law of Robustness via isoperimetry" by Bubeck and Sellke. Theorem 3. Let $\mathcal{F}$ be a class of functions from $\mathbb{R}^{d} \rightarrow \mathbb{R}$ and ...
user avatar
0 votes
1 answer
34 views

I am trying to understand this derivation

This might be a bit basic but, can someone tell me how did the authors go from the first equation to the next. Each case in the transfer set contributes a cross-entropy gradient, $dC/dz_i$, with ...
user avatar
2 votes
2 answers
53 views

Multiclass Classification: Why do we exponentiate the softmax function?

In the context of neural networks, we use the softmax output in multiclassification models. Firstly, let $P(y) = \sigma (z(2y-1))$, which comes from the definition of sigmoid units. We define $\bf z=\...
user avatar
  • 1,356
2 votes
0 answers
33 views

How "Optimal" are Solutions from Gradient Descent?

When optimizing the Loss Functions of Neural Networks using (some version of the) Gradient Descent algorithm, I have often heard this situation described as a Sequential Optimization Problem. This ...
user avatar
  • 1,700
1 vote
0 answers
23 views

Can Gradient Descent be "Combined" with Dynamic Programming?

In most applications of Gradient Descent (e.g. optimizing the Loss Functions of Neural Networks) - regardless of the "type" of Gradient Descent algorithm being used (e.g. Stochastic Gradient ...
user avatar
  • 1,700
0 votes
0 answers
7 views

approximating indicator function of threshold functions using 1 hidden layer of a neural network

I'm reading a paper about approximations using neural network and I'm trying to understand the next sentence (that isn't proven in the paper): I feel like I'm missing something simple here. The ...
user avatar
  • 189
0 votes
0 answers
13 views

Is there transfomation that transform the image to a space that I can use a small amount of function of fit it?

Now what I want to do is that I want to reprentation(just fitting) a image with neural network with as less error as possible,like paper NeRF(neural reprentation field) and FFN and Sire. The input is ...
user avatar
  • 41
0 votes
1 answer
39 views

How to find the size of an ϵ-net of a vector space?

Taken from paper "A Universal Law of Robustness via isoperimetry" by Bubeck and Sellke. Theorem 3. Let $\mathcal{F}$ be a class of functions from $\mathbb{R}^{d} \rightarrow \mathbb{R}$ and ...
user avatar
2 votes
0 answers
26 views

Parameter Estimation in a Gaussian Environment

In Simon Haykin's Neural networks and learning machines Page 106. This paper defines a function and intends to derive it to obtain the target result w. $$ \mathscr E(\mathbf w) = \frac{\sum_i^N(d_i - \...
user avatar
  • 21
0 votes
0 answers
11 views

Mini batches and loss in recurrent neural networks (RNNs)

Suppose that we have a sequence $\left\{x^{(k)}\right\}_{k = 1}^{N}$ and that we wish to use a RNN to predict the next element of the sequence given the previous elements of the sequence (e.g., a ...
user avatar
  • 303
0 votes
0 answers
9 views

Prior in variational autoencoders

I am currently dealing with variational autoencoders where I've read the original paper "An introduction to variational Bayes" from Kingma and Welling. I am currently still a little confused ...
user avatar
  • 53
0 votes
0 answers
30 views

Derivation in paper Deep Neural Networks as Gaussian processes in ICLR 2018

I am trying to understand the derivation of the main equation in the seminal paper titled Deep Neural Networks as Gaussian processes (in ICLR 2018). Following is the equation number (7), which can be ...
user avatar
2 votes
0 answers
88 views

How to grow the intuition behind this proof?

My intension is to understand this complete proof step by step in a lucid manner. I am trying few days to capture this, Unfortunatly I am failed. Can I hope to get a nice intuitive explanation of this ...
user avatar
1 vote
0 answers
28 views

Can a neural network with ReLU activation represents exactly all $B$-bounded and $L$-Lipschitz $K$-max-affine functions?

A max-affine function is defined as the maximum over a set of affine functions, which is always convex. More specifically, we define a $K$-max-affine function $f:\mathbb{R}^d\to\mathbb{R}$ that can be ...
user avatar
1 vote
0 answers
13 views

Universal approximation of neural networks

I am currently dealing with the topic of reproducing kernel Hilbert spaces (RKHS) given the draft book of Francis Bach. As a background knowledge for my current problem define: \begin{align} &H_1=\...
user avatar
  • 53
0 votes
0 answers
11 views

Why overparametrization is necessary if one wants to interpolate the data smoothly?

For my research work, I am reading this paper named A Universal Law of Robustness via Isoperimetry. Solving n equations generically requires only n unknowns. But I got stuck in this line However, the ...
user avatar
  • 257
0 votes
0 answers
10 views

What are the MATLAB syntax for various neural network learning algorithms and activation functions??

I just started learning MATLAB for neural network implementation. I just studied the various activation functions. However, I am confused in their syntax. Please help...
user avatar
1 vote
0 answers
15 views

Derivation of the integral form of the numerator in the Bayesian inference equation???? (not on the denominator)

In the reference Gaussian processes: iterative sparse approximations by Csató, Lehel (Csató, Lehel. Gaussian processes: iterative sparse approximations. Diss. Aston University, 2002), on page 20, ...
user avatar
  • 21
0 votes
0 answers
14 views

Confusion regarding the derivation of graph convolution

I am currently studying Spectral Graph Convolutions, and I am following this document: https://atcold.github.io/pytorch-Deep-Learning/en/week13/13-1/. They have derived the convolution as follows: The ...
user avatar
-1 votes
1 answer
53 views

Is it possible (in principle and in meaningful way) to describe any subset of n-dimensional real Euclidean space?

Let us start with some background and motivation. My main question is very simple and it is available few paragraphs further and it is written in bold. My problem is based from the emerging theory of ...
user avatar
  • 1,181
0 votes
0 answers
39 views

Understanding this Graph: What is a PetaFlop?

I was looking at this paper (https://arxiv.org/pdf/2005.14165.pdf) and came across this graph: I am trying to understand the following two things about this graph: What is PetaFLOP/s-days? I read ...
user avatar
4 votes
1 answer
132 views

Comparing the Training Costs of Machine Learning Algorithm: A Mathematical Perspective

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks. 1) Kernel Methods: For instance, I was looking at the optimization in ...
user avatar
  • 1,700
0 votes
0 answers
31 views

Notation to indicate input and output dimension of a function

I have quite a few functions $F_i:\mathbb{R}^n\to\mathbb{R}^m$, where $n$ and $m$ aren't the same for each $F_i$. e.g. $F_0:\mathbb{R}^2\to\mathbb{R}^2$, $F_2:\mathbb{R}^3\to\mathbb{R}^4$, etc. I'm ...
user avatar
  • 309
0 votes
1 answer
21 views

Isomorphic Neural Nets

The following is from Reconstructing a neural net from its output by Fefferman. In here, I'm not sure about the notation. Are we fixing one $l$ such that $\gamma_l$ is identity, and the rest of the ...
user avatar
  • 1,342
-1 votes
1 answer
81 views

Is "Probability Theory" an Inseparable Aspect of Machine Learning? [closed]

I have always had the following question about Probability and Machine Learning. As a simple example, suppose we have some data (e.g. heights of students: 175 cm, 181 cm, 162 cm, etc.) . If we assume ...
user avatar
  • 1,700
2 votes
0 answers
25 views

How come nonlinear optimization problems need careful choices of initial parameters but neural networks appear to not have this issue?

When I run some nonlinear optimization code - I often encounter people saying that there is no global nonlinear optimization code that is guaranteed to reach a global maxima. Instead it is recommended ...
user avatar
0 votes
0 answers
27 views

Derivative of L1-loss

If y is the ground truth label vector and ŷ the predicted label vector. Then ...
user avatar
0 votes
0 answers
14 views

Backpropagation through time example (Recurrent Neural Network)

can some one help me understand & calculate the back propagation through time algorithm for this example (ever values are just scalars): The unit has an input $x_t$, a hidden state $h_{t-1}$, ...
user avatar
3 votes
1 answer
90 views

Can Non-Convex Optimization Problems have Closed Form Solutions?

In the realm of statistical modelling, creating a statistical model with respect to some data involves optimizing some mathematical function (e.g. Loss Function, OLS Equation, Maximum Likelihood ...
user avatar
0 votes
0 answers
18 views

How can you convert a convolution stencil from a coarse to a fine mesh?

In the paper Learning Across Scales, the authors describe a multigrid method for transfering convolution operators to different meshes (pixel sizes in images); that is, given a 2d convolution kernel $...
user avatar
0 votes
0 answers
43 views

How to understand this gradient used here to compute the square root of $x$?

I found a snippet of C++ code to compute the square root of non-negative integer x via MSE Loss function and gradient descent. ...
user avatar
3 votes
2 answers
117 views

Do Neural Networks "Approximate Functions" or "Represent Functions"?

I was looking at the differences between these mathematical theorems: Universal Approximation Theorem (https://en.wikipedia.org/wiki/Universal_approximation_theorem) Kolmogorov Representation ...
user avatar
0 votes
0 answers
13 views

How to show a function with nonzero integral is discriminatory

This is a prove from the artile "G. Cybenko, Approximation by superposition of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989)", which uses the Wiener's Tauberian Theorem ...
user avatar
1 vote
0 answers
23 views

Approximation of continuous functions by neural network [closed]

For $f$ as (c) or (d), the answer is yes or no? I wonder that the nozero constant function in $[0,1]$ can not be approximated by such functions.
user avatar
2 votes
1 answer
52 views

Are Machine Learning Optimization Problem ever Categorized as "P" or "NP"?

In the context of Computer Science and Optimization, I have heard that different problems can be classified using the "P vs NP" framework. Essentially, there is a hierarchy of problems based ...
user avatar
  • 1,700
5 votes
1 answer
125 views

How can the Loss Functions of Neural Networks be Non-Convex?

I have heard the following argument being made regarding Neural Networks: A Neural Network is a composition of several Activation Functions Sigmoid Activation Functions are Non-Convex Functions The ...
user avatar
  • 1,700
2 votes
1 answer
48 views

Meaning of the Terms "Dense" and "Arbitrary": Approximation Theory and Neural Networks

I was reading about the mathematical background behind Neural Networks to try and understand why a Neural Network is in theory able to "work". I came across the following link: https://en....
user avatar
0 votes
0 answers
25 views

What is the derivative of a max function with 3 parameters?

I am not sure if this is considered a ReLU function but the function is, v0 = max(h0,h1,0) I need to find the derivative of v0 wrt to h0 and h1 (which are hidden ...
user avatar
2 votes
1 answer
50 views

Jacobian Matrix of an Element wise operation on a Matrix

From ref 1 it is clear that when you have an elementwise operation on a vector; the Jacobian matrix of the function wrto its input vector is a diagonal matrix For an input vector $\textbf{x} = \{x_1, ...
user avatar
1 vote
1 answer
62 views

Vector-matrix differentiation and vectorisation

In recurrent neural network backpropagation (BPTT), we have the equations: \begin{align} e_t &= E^T x_t \\ a_t &= W_{hx}^T e_t+ W_{hh}^T h_{t-1}\\ h_t &= \text{tanh}(a_t) \\ s_t &= W_{...
user avatar
  • 239
1 vote
0 answers
35 views

Matrix Derivation for Neural Network Formula

I am learning some insights of Neural network but I have some problem with the derivation of matrix for backpropagation. On an assumption that the formula for calculating for one node in a neural ...
user avatar
  • 239
1 vote
1 answer
41 views

A proof involving Batch-Normalization and SGD in Neural Networks

I am trying to understand a proof from this paper. Consider the following setting: We train a neural network layer with SGD, that is by updating the weights according to $$w_{t+1} = w_{t} - \eta \...
user avatar
  • 519
1 vote
0 answers
43 views

Why to normalize an adjacency matrix?

In Kipf & Welling (2017) paper https://arxiv.org/pdf/1609.02907.pdf. It uses the normalized adjacency matrix $\mathbf{A}_{symm} = \mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2}$. I know the largest ...
user avatar
  • 123
1 vote
1 answer
38 views

Interpretation of an undirected adjacency matrix

I am new and know not much about "graph theory" and "graph neural network". Assume, I have one incidence matrix $\mathbf{B}$ such as visitor item1 item2 item3 item4 A 1 0 0 1 B ...
user avatar
  • 123
0 votes
0 answers
17 views

Neural Networks in Mathematics

Neural networks are used extensively in machine learning and allied fields. I would like to know if neural networks crop up in an entirely non-machine learning setting. Such a setting should exclude ...
user avatar
1 vote
0 answers
20 views

Derivative of a Vector Function (Hinge) [closed]

I need to calculate the derivative of the Hinge loss function, but the formulation is quite unfriendly. Here's how it's been defined for an instance where the $i$-th index corresponds to the right ...
user avatar
0 votes
1 answer
43 views

Why does my regression-NN completely fail to predict some points?

I would like to train a NN in order to approximate an unknown function $y = f(x_1,x_2)$. I have a lot of measurements $y = [y_1,\dots,y_K]$ (with K that could be in the range of 10-100 thousands) ...
user avatar
  • 1
0 votes
0 answers
22 views

How to get a discrete direction from quaternion rotations

Say you have multiple quaternions (of the form q = w, x, y, z) that represent absolute rotations. How can you get a discrete direction from there such that if we see quaternion 0 for example we can ...
user avatar
-1 votes
1 answer
82 views

How $\Omega _X$ is related to $\Omega$ as subobject classifier - trying to understand toposes of deep neural networks

My aim is to understand how the notion of su-bobject classifier is used in the article Topos and Stacks of Deep Neural Networks about the categorical formalization of deep neural networks. This ...
user avatar
  • 1,181

1
2 3 4 5
15