Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

The maximal singular value or condition number of such random matrix product.

Let ${X}_i\in\mathbb{R}^{d\times d}$ denote $d$-by-$d$ random matrix. Every element of ${X}_i$ is sampled from Gaussian distribution $\mathcal{N}(0, 1/d)$ independently. Define ${R}_n$ as ($I_d$ is ...
0
votes
0answers
9 views

What scaling is appropriate for prediction intervals

I have a set of price forecasts which I am feeding in to a neural network for another prediction. Included in this price forecast are prediction intervals at 95, and 50% levels. What is the best way ...
0
votes
0answers
14 views

Details of proof of convergence of SGD

Deriving the SGD rule I came across some essential doubts about the stochastic gradient. In the proof that I'm reading, they introduce the subgradient $g$ together with the iteration rule as follows ...
2
votes
1answer
55 views

Variance of $\tanh z$ where $z\sim{\cal N}(0,\sigma^2)$

In Priors for Infinite Networks (Neal, 1996), the paper considers a simple one hidden layer neural network defined by \begin{align}h &= \tanh (a + Ux) \\ f &= b + Vh\end{align} where $f(x)$ is ...
0
votes
0answers
17 views

Universal approximation theorem for bag functions

Approximation Let $\mathcal{M}, \mathcal{T}$ be subsets of a topological space. $\mathcal{M}$ approximates $\mathcal{T}$ iff $\mathcal{T} \subseteq \overline{\mathcal{M}}$. Let $\sigma$ be ReLU. Let $\...
3
votes
0answers
35 views

Are there any good textbooks in recent years about the math of neural networks?

I'm looking into doing neural networks for my mathematics degree's student seminar, but I'm having a hard time finding a textbook to use that is more recent as this field is growing quickly, as well ...
1
vote
1answer
29 views

How can I approximate a sine function using neural networks?

I am trying to replicate the figure 5.9 of the book Machine Learning and Patter recognition. But in the figure the output function is a soft one, in my architecture it is just joined points. How can I ...
0
votes
1answer
55 views

What is the correct formula for updating the weights in a 1 hidden layer neural network?

I'm creating a neural network with 3 layers and no bias. On internet I saw that the expression for the derivative of the weights between the hidden layer and the output layer was: $$\Delta W_{j,k} = (...
0
votes
0answers
11 views

Statistical justification for monte-carlo dropout rate in neural network

Context: Monte-Carlo dropout is the process of randomly setting a number of units in a neural networks hidden layer to zero at test time. This makes the prediction process stochastic, so a cohort of ...
0
votes
0answers
20 views

Backpropagation: why partial derivative, not full derivative?

After studying backpropagation for neural networks, I have a question: why can't we use full derivatives for backpropagation? I understand why partial derivatives work in backpropagation. However I ...
0
votes
1answer
40 views

Batch gradient descent - matrix operations

I was trying to implement simple FNN network using tutorial on this page. So, given this super-simple network:                                          and if I use for training these 3 samples: \...
1
vote
1answer
51 views

Is there a closed form expression for entropy on tanh transform of gaussian random variable?

so my question is if there's a closed form (analytical?) expression for entropy for a variable $u$ defined as the $tanh$ of an gaussian random variable $x$. The reason I need an closed form solution, ...
0
votes
0answers
10 views

Is a Chi-Squared goodness of fit test appropriate for Neural Network regression?

So I always have wished that regression of neural networks gave more interpretable results and I'm pretty hopeful that chi-squared tests anchor these MSE values in the same way that accuracy anchors ...
0
votes
0answers
8 views

Solution of linear complementarity problems via Neural Nets

For a Linear complementarity problems (LCP) of the sort, with a given $Q$-matrix $\mathbf{M}$, $\mathbf{Mz}+\mathbf{q} \ge \mathbf{0}$ $\mathbf{z} \ge \mathbf{0}$ $\mathbf{z}^{\mathrm{T}}(\mathbf{Mz}+\...
4
votes
0answers
32 views

Statistical efficiency in context of neural networks?

According to https://en.wikipedia.org/wiki/Efficiency_(statistics), an estimator with statistical efficiency is one that "... needs fewer observations than a less efficient one to achieve a given ...
0
votes
0answers
17 views

Equation (11.12) in Hastie, Tibshirani and Friedman's Elements of Statistical Learning

I am reading Hastie, Tibshirani and Friedman's Elements of Statistical Learning. Equation (11.12) on page 396 (Section 11.3 Neural Network) says $$ \frac{\partial R_i}{\partial \beta_{km}} =-2(y_{ik}-...
0
votes
0answers
11 views

Problem with understanding residual network variance analysis

I try to understand the analysis of variance and mean in a deep residual network. In this article, on the second page, why we can write Var(x_i^{l+1})=Var(x_i^l)+Var(f_i^l(x_l))? Are they independent ...
0
votes
0answers
8 views

How does iterative input to a recurrent linear associator mathematically produces the correct response?

I am reading neural networks and I am now studying Recurrent Linear Autoassociator. I am stuck at its mathematical working. Let us assume we have a nonsingular symmetric matrix $R$ whose Eigenvectors ...
-1
votes
1answer
28 views

variance of inner product of vectors sampled from gaussian distribution? [closed]

Suppose that we have 2 vectors such W and X $\in R^N$ that each element sampled from normal distribution $\mathcal{N}(0, 1)$, and the inner product $Z=W^T X$. Why is variance $V[Z]$ equal to $N\sigma^...
0
votes
0answers
22 views

Formal robustness verification of neural networks: MILP vs SMT

I'm not sure if this is the right place to ask this questions. I'm working my way into the field of formal verification of neural networks. The goal is to analytically evaluate the robustness of ...
1
vote
1answer
63 views

Calculating derivatives with respect to a weight matrix

I am currently doing a machine learning course and I am trying to wrap my head around back propagation. I watched these videos which helped clear things up a bit. I am trying to apply the same ...
3
votes
1answer
63 views

Neural network architecture capable of performing a sum by category?

I am wondering whether it would be possible to build a NN that can be trained to take 2D training examples (with a fixed number of rows) where the two columns would represent an amount and a category, ...
0
votes
0answers
14 views

Gradient norm in a neural network is bounded?

Consider a fully connected neural network with single hidden layer $f(x,w) = w^T_2 \sigma(w^T_1 x) $ where $w = [ w_2, w_1 ]$ are networks' parameters and $\sigma$ is an activation function (e.g tanh, ...
0
votes
0answers
31 views

Context: “Squashing Functions and Neural Networks”

I hope your lives have been proceeding along well. Lately, I have been reading about squashing functions in the context of neural networks. Specifically, the book I am working through, Deep Learning ...
1
vote
1answer
31 views

Computing the gradient of a two-layer neural network w.r.t. the data

I want to verify that I am computing the following gradient correctly. I am looking at this conjecture, and the gradient is paramount. Using the definitions from the the paper, consider the data $\vec{...
0
votes
1answer
15 views

Gradient of 2 layer network

I am considering a simple $2$-layer network with $m$ training data $(x^i, y^i)$ data whose cost function is $$\ell(w,\alpha, \beta) := \sum_{i=1}^{m} \left( y^i - \sigma(w^Tz^i) \right)^2$$ where $$\...
0
votes
0answers
17 views

Can more data hurt the train and test losses in deep neural network?

I would really appreciate it if someone can answer my following questions. 1- Does the "double-descent" concept mean that if I make the size of the neural net bigger than the interpolation ...
1
vote
0answers
25 views

Linearize a sign function

Let's say I have some matrix, $M$, of real numbers, and I apply the $\text{sign}$ function to it, such that $m_{i,j} > 0$ gets mapped to $+1$ and $m_{i,j} \le 0$ gets mapped to $-1$. Let's call ...
0
votes
1answer
22 views

Obtain non linear solution using neural network

The function f(x)=theta·x where theta is a row vector and x is a column vector, is a linear function. How can I obtain a non-linear function g(x) using a multi layer network, that also takes in x as ...
2
votes
0answers
54 views

What is The Correct Formula For Neural Network Gradient Descent?

Given a neural network with an activation function $f(x) = \frac{1}{1+e^{-x}}$, I calculated the gradient of any weight $W$ at layer $l$ where the activation sum of the layer is $A$ in the index $i,j$ ...
1
vote
0answers
78 views

Am I Calculating The Partial Derivative Of Neural Network Cost Function Correctly?

I have been trying my best to understand how gradient descent works. Here I have some scratch notes from when I was attempting to derive a formula. Ignore the subscripts of the matrices. My work ...
0
votes
0answers
23 views

How do I know an ensemble of feed forward neural networks with different activations is a universal approximator?

Basically I took a look at this paper and it is clear that, "A standard multilayer feedforward network with a locally bounded piecewise continuous activation function can approximate any, ...
0
votes
1answer
66 views

When is it OK to introduce multicollinearity? (Machine Learning)

Looking at the Kaggle Titanic Dataset there exists two columns of predictors: Parch describes the sum of that passenger's parents and children that have boarded <...
1
vote
0answers
13 views

How is GRU outputting the following Values? I am getting a different answer<Keras>

When I run the following code I get this ...
0
votes
0answers
14 views

Stacked RNN BPTT gradient calculation

In an vanilla LSTM model, the BPTT in an individual cell consists of the formulas: The formula is from : https://medium.com/@aidangomez/let-s-do-this-f9b699de31d9 $\delta$outt = $\Delta$t + $\Delta$...
1
vote
0answers
50 views

Derivative of Binary Cross Entropy for Neural Network Classifier

I am currently following a introductory course in machine learning. I would like to use the binary cross entropy as a loss function. Having a simple neural network (2 inputs, 1 hidden layer with 3 ...
1
vote
0answers
26 views

Can Euler's totient function be made continuous enough for an ANN?

I am way out of my comfort zone here, so please be gentle! I have been reading papers that try to use Artificial Neural Networks to approximate the Euler's Totient function of form $\varphi(n) = (p-1)(...
1
vote
0answers
31 views

Is it possible to calculate the feed forward Hessian inverse?

Did I made a calculation error? Say we had a simple one layer perceptron where: $f$ is the activation function, $w$ is the weights matrix, $b$ is the bias vector, $x$ is the input vector, $y$ is the ...
0
votes
0answers
18 views

Making a well-informed guess about the form of an ODE based on an image of its vector field using intuition and/or neural networks?

I haven't worked with ODEs in a long time so I don't have great intuition in this area. Is it possible to make a well-informed guess about the form of an unknown ordinary differential equation based ...
0
votes
0answers
13 views

Reinforcement learning DQN environment structure

I am wondering how best to feed back the changes my DQN agent makes on its environment, back to itself. I have a battery model whereby an agent can observe a time-series forecast of 17 steps, and 5 ...
1
vote
1answer
72 views

Neural ODE definition of derivative $\frac{d L}{dz(t)}$ (adjoint)

The authors of Neural Ordinary Differential Equations (NeuRIPS 2018 best paper award), propose to model machine learning problems with an ODE $$ \dot z(t) = f(t, z(t), \theta) \quad{\text{s.t.}}\quad ...
0
votes
2answers
49 views

Row-wise derivative of a vector with respect to a matrix

Consider a column $n\times 1$ vector $\overline {z}$ and an $n\times m$ matrix $W$. What would one call and denote an $n\times m$ matrix of derivatives defined by $M_{ij}=\frac{\partial z_i}{\partial ...
1
vote
0answers
19 views

Partial-derivative in an artificial neural network. Product of vectors of different length?

Introduction I am programming an artificial neural network to analyze the MNIST dataset of handwritten digits. Vector $\textbf{a}$ in layer $\textit{L}$ of length $\textit{i}$ in the network are given ...
1
vote
0answers
35 views

Problems in getting the derivatives of the batch normalization layer

I'm working on understanding the math used in the batch normalization layer in the CNN and found the original paper discussing this trick: Batch Normalization: Accelerating Deep Network Training by ...
0
votes
0answers
110 views

Are Resnet Neural Networks non-affine transformations with linear activation functions

Given an Artificial Neural Network that allows "skip" connections or inter-layer connections such as found in Resnet, is it possible to get non-affine mappings of the input to the output ...
0
votes
1answer
52 views

How can the $\tanh(x)$ activation function be written for two variables? $\tanh(x,y)$

The tanh activation function is: $$\tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Is there a corresponding formula for variable pairs? $$ \tanh(x,y) =?$$ My uneducated guess is that ...
0
votes
0answers
21 views

Accurate metric around zero for neural networks

I´m trying to approximate a continuous function via regression with a neural network. The function takes values in the interval $[-10, 10]$, but the values around zero are more important, because, in ...
1
vote
0answers
28 views

Dirichlet Energy for Graphs, Derivation

I would like to prove this formulation of the Dirichlet Energy for Graph Neural Networks $$ \begin{aligned} E(\mathbf{X}) &=\frac{1}{d_{i}} \sum_{j \in \mathcal{N}(i)} w_{i j}\left\|\mathbf{x}_{i}-...
0
votes
0answers
17 views

Gradient of a magnitude of matrix product

I'm trying to solve one optimization using a neural network. Let's say, I have a neural network, i.e., a function $f$ operating on a matrix and outputs another matrix. Two things are given: $\nabla f$...
0
votes
0answers
7 views

What would be the suitable neural network architecture to start with for huge output label classification task?

I have huge output classes(around ~70k), What would be the suitable neural network architecture to start with for this huge output label classification task?. I'm thinking of converting these 70k ...

1
2 3 4 5
13