Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Change in classwise distribution of hidden layer outputs given categorical crossentropy loss in a single layer linear neural network

Given a linear neural net with a single hidden layer, and a set of input samples $\mathbf X\in\mathbb R^{m\times n}$. Consider an input $\mathbf x\in \mathbf X$, such that output $$\mathbf z=\mathbf{...
Phoenix's user avatar
  • 29
0 votes
0 answers
23 views

How does the ReLU function introduce non-linearity into Neural Networks?

Can anyone explain how the ReLU function introduces non-linearity into a neural network? I've been told it does, but the outputs of $f(a) = max(0, a)$ seem either like the neuron is 'inactive' (i.e. $...
M. W. Howells's user avatar
0 votes
0 answers
6 views

Do I need model calibration/feature normalization for reliable path integrated gradient/sampled Shapley feature attribution in a dnn model? [migrated]

Are model calibration and feature normalization required for path integrated gradient and sampled Shapley-based feature attribution analyses to work properly in a deep neural network model? I read ...
jjwest's user avatar
  • 1
5 votes
1 answer
80 views

Rigorous Mathematical foundations of Machine Learning / Deep Learning / Neural Networks

I am an Engineering Graduate (with a strong background in Probability/Measure Theory, Linear Algebra and Calculus) wanting to dig deep into Deep Learning and Neural Networks, and I'm looking for ...
Michel H's user avatar
  • 300
0 votes
0 answers
25 views

Stochastic gradient descent with momentum: eigenvalues

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in particularly one point: I am trying to derive the convergence rate for ...
Patricio's user avatar
  • 567
1 vote
1 answer
73 views

What is the importance of the largest eigenvalue / spectral radius of a symmetric positive matrix being equal to 1? Particularly in attention.

It is often said, that if the spectral radius of a matrix $\boldsymbol{A}$ is equal to $1$, the matrix has "regularizing" properties for the matrix product $\boldsymbol{Ax}$ for a vector $\...
Philipp123's user avatar
1 vote
1 answer
58 views

How would one formalize the mathematical operations of Tensors in deep learning libraries (TensorFlow, Pytorch, Numpy)?

I'm having trouble in understaning how to formalize some of the operations done in libraries such as Numpy, TensorFlow, Pytorch and others. From what I know, some basic operations such as summing are ...
filip augusto's user avatar
-2 votes
0 answers
13 views

Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks [migrated]

I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
John Title's user avatar
1 vote
1 answer
161 views

Proving Density for Function Approximation with Hidden Layer Perceptron

I'm working on a problem related to function approximation within the $L^2\left(I_n\right)$ space of square-integrable functions: Problem Statement: Given a lemma without proof: $\textit{Lemma}$: Let $...
BlizzardWalker's user avatar
1 vote
0 answers
20 views

How to comment on goodness of loss functions?

I have two loss functions $\mathcal{L}_1$ and $\mathcal{L}_2$ to train my model. The model is predominantly a classification model. Both $\mathcal{L}_1$ and $\mathcal{L}_2$ takes two variants of the ...
Aleph's user avatar
  • 163
0 votes
1 answer
79 views

Non-Dimensionalizing a Traffic Flow PDE for Physics Informed Neural Network Issue

I'm working on analyzing a traffic flow model described by the following partial differential equation (PDE): $V_{\max} \left(1 - \frac{2\rho}{\rho_{\max}}\right) \frac{\partial \rho}{\partial x} + \...
Proxy's user avatar
  • 3
1 vote
0 answers
13 views

ANFIS Model, tips about improving performance [closed]

I have a question regarding about improving the performance of an ANFIS (adaptive neuro Fuzzy inference system) model. In MATLAB, I have been training a model with 5 inputs, with 816 data point for ...
jocelyn matus ancavil's user avatar
1 vote
0 answers
34 views

Expression for derivative of neural net output w.r.t edge weight

This is a question on the mechanics of backpropogation derivatives in NNs. I've seen plenty of analysis on the derivative of a NN output with respect to a given input, but am unsure on how to compute ...
redbull_nowings's user avatar
1 vote
0 answers
300 views

Calculating $\min(x_1,x_2)$ and $\max(x_1,x_2)$ using a two-layer neural network

Suppose that $\vec{x}\in\mathbb{R}^2$ is a vector and we want to find the minimum and the maximum of its components, using a two-layer neural network: $$\vec{y} = f_2(W_2f_1(W_1\vec{x}+b_1)+b_2), \ \ ...
S.H.W's user avatar
  • 4,379
0 votes
2 answers
138 views

Using Matrix Calculus in Backpropagation derivation. Rules of order of matmul and transposition when taking derivatives in different layouts.

I define a neural network with $L$ layers ($L-1$ hidden layers). The forward pass is as follows: $$ \mathbf{a}^{(l)} = f(\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}) $$ Where $l \in [0,L]$ and ...
xTom's user avatar
  • 45
0 votes
0 answers
18 views

Gaussian Kernel outputs a 2*m feature map?

I am currently writing my masters thesis on the Double Descent Curve in Neural Networks and as I was doing some research, I came across the paper "On the Double Descent of Random Features Models ...
Silvio's user avatar
  • 21
0 votes
0 answers
66 views

Posterior probabilities in a GMM

This is a statistics/probability question formulated in the context of machine learning (problem 6.17 in Bishop's 'Deep Learning' book). We are modelling the conditional distribution $p(\mathbf{t}|\...
Mat Dyl's user avatar
  • 511
0 votes
1 answer
58 views

Could someone give me a concrete mathematical definition of a "linear translation"?

I'm analyzing a paper titled "Distributed Representations of Words and Phrases and their Compositionality". The paper deals with Natural Language Processing, so it has math concepts involved ...
Eliza Romero's user avatar
1 vote
1 answer
82 views

Exercise 5.26 of PRML By Bishop

This is the question: Equations 5.127 and 5.128: I am stuck on proving 5.206. This is my proof, and I am getting the wrong answer: $\Omega_{n} = \frac{1}{2} \sum_{k} \left( \sum_{i} J_{nki}\tau_{ni} ...
sage76's user avatar
  • 41
0 votes
0 answers
28 views

closed loop operator for neural networks, final form.

I'm reading the book about neural networks from Simon Haykin. In the first section of recurrent neural networks plot this figure: and he says that the system is linear. So he discribes the system in ...
João Paulo Andrade's user avatar
0 votes
0 answers
32 views

What are RNN trying to do from a statistical point of view?

I know how recurrent neural networks work but let's say that I want to model their behaviour from a statistical point o view, how should I interpret their output? Surprisingly on the internet I found ...
Francesco De Santis's user avatar
1 vote
1 answer
95 views

Neural ODEs-Dimension Mismatch in the proof of adjoint sensitivity method

In the paper 'Neural Ordinary Differential Equations' (2019) by Ricky T. Q. Chen et al., it is noted that adjoint variable $\mathbf{a}(t)$ as the gradient of the loss function with respect to the ...
CKTPU's user avatar
  • 19
0 votes
1 answer
53 views

Stochastic/sequential gradient descent for the perceptron

I'll quote Bishop's Pattern Recognition and Machine Learning (2006) in the context of the perceptron, an algorithm that, given two finite subsets of "patterns" $\mathcal{C}_1,\mathcal{C}_2\...
Sebastián P. Pincheira's user avatar
0 votes
1 answer
51 views

Manually setting up integral weights of a 2-2-2 neural network.

inputs and outputs are both binary. hidden layer and output layer have biases. function to be modelled is f(i1, i2) = 2 - i1 + i2 transfer function is sigmoid at hidden and output layers, and the ...
aryan's user avatar
  • 9
0 votes
0 answers
100 views

Cut the circles in a way that would be similar to already cut circles.

[1]:https://i.stack.imgur.com/0aien.jpg [2]: https://i.stack.imgur.com/x9fAv.jpg So I have this two images as you can see in [2] image circle is cutted from above and below. I have data lots of [1] ...
unit 1991's user avatar
  • 879
3 votes
2 answers
98 views

Can MLPs represent functions exactly for finite inputs?

The universal approximation theorem states given appropriate depth / width, an MLP can represent any continuous function with arbitrary precision $ \epsilon > 0 $. For discrete functions, $f: \...
rossignol's user avatar
0 votes
1 answer
51 views

What am I misunderstanding in a paper using the Abel's functional equation?

I'm currently reading the paper available at https://arxiv.org/abs/1503.05724. In this paper, the authors define a function iteration as shown here: . They also discuss Abel's functional equation: . ...
Naetmul's user avatar
  • 282
2 votes
1 answer
87 views

Calculating the derivatives in a linear layer in NN

There is the following puzzle that stems from Neural networks: I have a matrix $\mathbf{Y} = \mathbf{X}\mathbf{W}^{T} + \mathbf{B}$ where $\mathbf{Y} \in \mathbb{R}^{S \times N}$, $\mathbf{B} \in \...
Jose Ramon's user avatar
1 vote
1 answer
93 views

Recurrent neural networks stability

I'm reading some papers on stability of neural networks mainly a dynamical system point of view. RNN can be thought of as $h_t=f(h_{t-1},x_t,\theta)$ where $\theta$ represent some parameters that are ...
user1880062's user avatar
1 vote
0 answers
38 views

Complex valued neural networks vs 2 channel real imaginary real-valued networks

Complex valued neural nets (CVNNs) use complex number natively and Wirtinger calculus for computing gradients, as I've read. I have trouble finding an explicit explanation of the clear difference ...
AlanTuring's user avatar
0 votes
0 answers
20 views

Why I can numerically use signatures on rough paths?

In the paper deep signature transforms ({bonnier, kidger, perez, salvi, tlyons}) they use the signatures on Brownian motion and they invert it with the inversion method defined in (The insertion ...
Omer's user avatar
  • 374
1 vote
1 answer
61 views

Analytical formula for third-order partial derivatives of feed-forward neural networks

is there an analytical formula for the third-order partial derivatives of a simple feed-forward neural network's output with respect to its inputs? (NOT with respect to the weights and biases of the ...
LockenLui's user avatar
0 votes
0 answers
20 views

Question about Long Short-term memory neurons in recurrent neural networks

This is the schematic for an LSTM neuron, given in Wikipedia I have question on output gate. It passes the cell state trough tanh function, which maps the value to an open interval between -1 and 1. ...
Mr. Proper's user avatar
0 votes
0 answers
69 views

Why is Gradient Descent always chosen for Neural Networks?

I am trying to understand why Gradient Descent are the chosen types of algorithm for optimizing the Loss Function in Neural Networks - and why other algorithms (e.g. EM Algorithm https://en.wikipedia....
stats_noob's user avatar
  • 3,206
3 votes
1 answer
51 views

Searching for Functions Exhibiting Semigroup Property for Energy-Efficient Neuronal Modeling

I am working on designing energy-efficient neurons for neuromorphic computing. One of the critical aspects of the dynamics I'm exploring is that they should adhere to the semigroup property. ...
Andi Faust's user avatar
2 votes
0 answers
41 views

Quantitative approximation by shallow ReLu networks

Let $f: I \to \mathbb R$ be an $L$-Lipschitz function with compact support. Show that for all $\epsilon > 0$ there is a function $\phi$ given by $$ \phi(x) = \sum_{i=1}^W c_i \mathrm{ReLu}(a_i x + ...
Hölderlin's user avatar
1 vote
1 answer
51 views

How to write chain rule when outputs are vectors

Consider the following machine learning problem: We have input matrix $X_{d \times N}$ and output matrix $y_{o \times N}$, where $N$ is the number of samples, $d$ is the input dimension and $o$ is the ...
KRL's user avatar
  • 1,180
0 votes
0 answers
34 views

Maximize auto-difference by sine transformations

The title is somewhat unclear because I do not know the words to describe this. Given $n \in \mathbb{N}^*$ , I want to find the real valued coefficients $(a_i, b_i), i \in [|1, n|]$ that minimize $...
Yeb02's user avatar
  • 11
1 vote
0 answers
32 views

Neural Network Linear Activation Functions

I understand the intuition that the sum of linear functions is again linear, and that is why a neural network with linear activation functions yields a linear model. But what I'm confused about is ...
PerplexedPelican's user avatar
0 votes
1 answer
80 views

Choose topic and direction for PhD (NN as universal approximator PDE Equations) [closed]

The problem arose of choosing a direction for studying and working within the framework of this topic: neural networks as universal approximators for solutions of partial differential equations. At ...
Alex 's user avatar
  • 225
1 vote
1 answer
48 views

Questions regarding backward propagation

I am reading article regarding backward propagation https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ . Lets say if I follow the example in the article but using only 3 nodes, ...
CKT's user avatar
  • 15
0 votes
0 answers
20 views

What an Implicit Neural Representation

I read papers about Implicit Neural Representation or Cordinate-based MPL. In these papers, they state that "this kind of MLP has the capability of learning how to map the coordinate to its value....
M Ali's user avatar
  • 1
2 votes
0 answers
63 views

How can an algorithm for traveling salesman beat concorde?

I am trying to learn about neural networks. I was reading the paper An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem which uses graph neural networks such that ...
edamondo's user avatar
  • 1,297
1 vote
0 answers
46 views

Coefficient for the gradient term in stochastic gradient descent (SGD) with momentum

I'm studying SGD with momentum and have come across two versions of the update formula. The first is from a wiki same as from the original paper: $$ \Delta w^t = \alpha * \Delta w^{t-1} - lr * \nabla ...
W Lewis's user avatar
  • 23
0 votes
0 answers
26 views

Relation between expectation and risk minimization

Let $`\mathcal{X}$ be the random vector describing the input distribution of the NN, and $\mathcal{Y}$ be the output distribution. Let, X, Y be the finite samples we have that form the training set (...
user1017330's user avatar
0 votes
0 answers
41 views

Perceptron Convergence: Monotonically Approach Solution?

I'm new to learning about perceptrons, but I saw a proof (for perceptron for binary classification with the caveat of forcing the separator through the origin) that, assuming the data is linearly ...
mishar's user avatar
  • 101
3 votes
1 answer
151 views

How do details of the proof of the universal approximation theorem work?

So I have a proof of the universal approximation theorem and need to understand how it works. So this is the theorem: Let $ d \in \mathbb{N} $, let $K \subseteq \mathbb{R}^d $ be compact, and let $\...
Lopsio's user avatar
  • 83
-2 votes
1 answer
124 views

A brilliant introductory course on machine learning (mathematical perspective) (simulation + implementation) [closed]

I am a grad student with a relatively good understanding of stochastic analysis / probability theory, but only basic coding experience. What is a good source (textbook or lecture notes) for an ...
rogerroger's user avatar
0 votes
2 answers
110 views

What are the formulas for ReLU, Tanh, and sigmoid activation functions [closed]

So I’m studying neural networks, and I came across these activation networks. Of course I could just ignore the mathematical specifics but they seemed interesting, especially since I like math. So ...
Anirudh Menon's user avatar
1 vote
0 answers
144 views

Mathematical efforts to embed any object into n-dimensional Euclidean space (or spaces)?

There are efforts to automate proof discovery using deep neural networks. There are varied approaches how the mathematical objects can be embedded into the layer of neural networks. E.g. one can ...
TomR's user avatar
  • 1,313

1
2 3 4 5
17