Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

From derivatives to gradients in backpropagation

Consider the following explanation of backpropagation from Wikipedia: Given an input–output pair {\displaystyle (x,y)}(x,y), the loss is: $ C(y,f^{L}(W^{L}f^{L-1}(W^{L-1}\cdots f^{1}(W^{1}x)\...
0
votes
0answers
21 views

Non Linear Activation functions in Neural Networks

I have three questions regarding non linear activation functions in Neural Networks. 1) Activation functions are used to introduce a non-linearity in a neural networks allowing them to model more ...
1
vote
0answers
27 views

Understanding this derivation of back propagation

I'm following along and trying to prove the 4 equations for back propagation in neural networks in chapter 2 of this book: Prove: \begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \...
0
votes
0answers
7 views

Using output bias to enforce the boundary (initial) condition in optimization with neural networks

Consider Neural Networks $\mathcal{NN}$ as functional approximators of say $\mathscr{C}_{x_0}([a,b])$ continuous functions over $[a,b]$ with initial value $x_0\in\mathbb{R}$. Since regularization ...
0
votes
2answers
17 views

Chain rule anomaly when applied to the backpropagation algorithm for neural networks?

This is a question that arose from watching 3blue1brown's video on the explanation of backpropagation in neural networks using calculus. I am familiar with the chain rule (for some context, I was ...
0
votes
1answer
25 views

If a neural network is a function f that maps x to y, how can I formally define a neural network with multiple outputs?

Formally I can say that a simple neural network can be formally defined as where D is the size of the input vector x and L the size of the output vector y. So I can say that $y = f(x)$. But how do I ...
0
votes
0answers
13 views

Independence Problem between CNN Weights and Gradients

It's said in many papers like that 'the gradients $g$ are independent from the weights. These assumptions hold for linear networks.' in Understanding the Effective Receptive Field in Deep ...
0
votes
0answers
24 views

Stock prediction

I'm working on a stock prediction algorithm using LSTM network. However my test data are obviously from different distribution than my train set 1, hence my prediction looks like the one on the figure ...
0
votes
2answers
32 views

Adjusting the Weight Matrix in gradient Descent backpropagation through neural networks

In many gradient descent algorithms to backpropagate an error through a neural network the final line looks something like this: $$ W_{ij} = W_{ij} - \mu \frac{\delta E}{\delta W} $$ i.e. adjust the ...
0
votes
0answers
13 views

Derivate of cost function by argument of activation function

On this link i found a paper written by Xavier Glorot and Yoshua Bengio that explained why training DNNs was so difficult in the beginning. First 5 pages are mostly empirical. They explain how they ...
0
votes
0answers
8 views

What's the derivative of a recursive function $ H_{t} = tanh(Wh * H_{t-1})$ with the product rule involved

I'm writing my own recurrent neural network and it's known to use backpropagation through time. In this, I have a weight $ Wh $, which gets used multiple times in a funtion $ H_{t} $ like in this ...
-2
votes
0answers
26 views

Use neural network to create a difference equation?

Got this idea. Assume that we have a output data $y (k) $ and input data $y (k) $ and we want to create an difference equation. $$ay (k) + by (k-1) + cy (k-2) + \dots + Q_n y (k-n) = du (k) + eu (...
1
vote
0answers
16 views

Finding correlation between function input and output

I'm trying to get a value for a correlation between a function input and its output. One brute force way to get this is to sample the entire space and find the standard deviation of the resulting ...
0
votes
0answers
4 views

What is the gradient of Q-value W.R.T policy parameters?

I have been recently studying Actor-Critic algorithms, and I ran into this question: Let $Q_{\omega}$ be the critic network, and $\pi_{\theta}$ be the actor. It is known that in order to maximize the ...
0
votes
0answers
11 views

Derivation of partial derivatives regarding Xavier initialization

I'm reading the paper on Xavier initialization (Understanding the Difficulty of Training Deep Feedforward Neural Networks (Glorot and Bengio, AISTATS 2010)) and had a question regarding the derivation ...
0
votes
0answers
16 views

Is there a way to use Artificial Neural Network to identify the primary cause(s) for the resultant value?

Suppose I have a set of data ( attached ) and one column is viewed as consequence ( e.g. wage per hour ) while otheres are considered as potential factors for now. My task is to find out what assumed ...
1
vote
0answers
18 views

Why are important equations in data science squared as the final operation?

Apologies for the naivety of this question, but as I begin my data science education I wish to develop a deeper intuition regarding the associated math. The cost function used in neural networks ...
0
votes
1answer
30 views

Write equation of a simple neural network

I have a number of weights and biases, and would like to write the network equations explicitly to understand how this works. $$w^2 = \begin{bmatrix} 0.336067 & -0.322224 \\ -0....
0
votes
0answers
9 views

Does padding data with zero affect gradients?

If we have a bunch of variable length sequences and pad them with 0s to all have the same length, so we can do minibatching, and then directly pass this minibatch into a 1D convolution, will the zeros ...
0
votes
0answers
32 views

Relationship between transpose and derivative of a matrix

This feels like a dumb question, but I am reading a paper where there is a function defined $x_t=W_{rec} \sigma (x_{t-1}) + $ (some other stuff) and then it says that for any $x_t, x_k$, $$\frac{\...
0
votes
0answers
32 views

Backpropagation derivation in Neural Networks

I am studying backpropagation in Neural Networks and I'm currently looking at the following video (not needed to answer the question): https://www.youtube.com/watch?v=GlcnxUlrtek&t=29s There is ...
1
vote
0answers
27 views

Why linear activation function fails the universal approximation theorem for neural network

I understand what UAT is and how it holds true for sigmoid and RELU activation functions. I've seen enough articles and explanation which visually explains how Sigmoid/Relu activation units are used ...
0
votes
0answers
25 views

Backpropagation derivation- chain rule expansion

I'm trying to write out the calculations for backpropagation but I'm having trouble getting the final answer- I believe I should be getting something similar to $-(y - \sigma(w \cdot x + b))\sigma'(w \...
1
vote
0answers
9 views

Central limit theorem on linear combination of activations in a neural network

While I've read a paper, Deterministic variational inference for robust bayesian neural networks, I have some confusion parts. On the contrary to a standard neural network framework, the author ...
2
votes
1answer
35 views

Method of Adjoints, Neural ODEs

I've been trying to understand the gist behind the Chen et. al paper on neural ODE's (https://arxiv.org/pdf/1806.07366.pdf). It seems like the main trick here is to be able to take derivatives of ...
0
votes
0answers
20 views

Feedforward neural networks: how to obtain the gradient of the loss function with respect to the weight matrix of the second to last layer

I have recently run into this page. I am trying to obtain the equation labeled BP4 by the author. Particularly, I want to obtain BP4 for the case of a feedforward neural network consisting of three ...
0
votes
1answer
20 views

Finding the coefficients of a piecewise linear function in a different basis

Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions ...
0
votes
0answers
24 views

Calculating a weighted sum with matrices, as used in neural networks.

I have been working with neural networks for some time without employing matrices. So for example, if you have an array of $n$ inputs $X_i$ to a neuron, each feeding that neuron via a connection with ...
0
votes
0answers
19 views

How can I normalize my input data?

f(x) and σ(·) is a custom activation function where $x $ is a $ D$ dimensional input vector, w is a $ D$ dimensional weight vector and b is a scalar My $x$ data is negative or positive between $-1 $...
0
votes
0answers
21 views

Counting the width and depth of a ReLU FNN implementation

I am trying to understand the way that the writer is counting the width and depth of a ReLU feedforward neural network (FNN) that we implement in the proof. In this paper, we define the width of a ...
0
votes
0answers
17 views

Neural network in Credit risk

I study Machine Learning and I'm interested in Credit Risk's subject. I use German Data set from https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+data). Where there are 20 attributes. On ...
3
votes
0answers
93 views

Approximation of positive definite functions by neural networks

Bochner's theorem shows that probability measures $\mu$ are linked with positive definite functions via Fourier transform: $f(k) = \int_{\mathbb{R}^n} e^{-2 \pi i k x} \,d\mu(x)$ Currently, ...
0
votes
0answers
17 views

Difference between local multivariate optimization and stochastic gradient descent?

Sorry if this sounds like a basic question but I don't understand what is the difference between local multivariate numerical optimization (minimization) and ...
1
vote
1answer
29 views

derivation of the cross-entropy cost function

I'm reading a popular book on neural networks, here's the link. http://neuralnetworksanddeeplearning.com/chap3.html In the excerpt below $C$ denotes the cost function, $b$ - bias, $w_i$ - $i$-th ...
1
vote
0answers
42 views

Covariance of a rectified (relu) Gaussian

Given a normal random vector $$X\sim N(\mu,\Sigma)$$ for spd $\Sigma$, I'm interested in the covariance matrix $K=\mathrm{cov}(Y)$ of the variable $$Y = \mathrm{relu}(X)$$ where the relu is performed ...
-1
votes
1answer
116 views

Convergence Rate and Complexity of Single SGD update for Neural Networks

Setting: Fix positive integers $m_1,\dots,m_n$ an activation function $\sigma:\mathbb{R}\rightarrow\mathbb{R}$ be $C^1$ with Lipschitz derivative, and let $NN$ denote the set of all feed-forward ...
0
votes
0answers
21 views

Derivation of winograd filter transform matrices

For http://web.archive.org/web/20190509195948/https://www.intel.ai/winograd-2/ , how to derive the winograd filter transform matrices ?
0
votes
0answers
26 views

Upper bound for the variance of an output neuron of a neural network

Modern neural networks oftentimes apply the softmax operation to their output neurons which is defined by $$s(x)_i = \frac{e^{x_i}}{\sum_{j=1}^N e^{x_j}}$$ Here, $i$ is the index of an output neuron,...
0
votes
0answers
21 views

boundedness of elements of a converging series

The background of my question is that I am trying to study properties of neural networks. In this analysis, I ran into the following question: Consider the following function $$ f_n(x) = \frac{1}...
0
votes
0answers
12 views

Deep learning: Model Performance when adding more input features

assume the train and validation loss of a NN converge to about the same value. Is the following statement true in general? (No more information given): When I add more input features, the performance ...
0
votes
0answers
15 views

Why $p(x|z)$ is assumed to follow multivariate Gaussian distribution in Variational Autoencoder?

In VAE paper including Kingma's paper, they assumed $p(x|z)$ to be a Gaussian/Bernoulli distribution according to data type. $p(x|z)$ as a decoder functions to map the latent value $z$ to the ...
0
votes
1answer
25 views

Is linear map the only one that is invariant under composition?

The composition of linear maps is clearly linear. Suppose a funtion $f:A \mapsto B$ and it belongs to some certain function class $\mathcal{F}$: $f\in\mathcal{F}$. Does $f\circ f\in\mathcal{F}$ $\...
1
vote
0answers
18 views

Parametric Continuous Convolution Layer: Formula explained

while reading the Deep Parametric Continuous Convolutional Neural Networks paper, I struggled to understand the formula for their parametric continuous convolution layer (page 2) : $$ \large{ h_{k,i} ...
1
vote
0answers
20 views

Got stuck at trying to figure out what the single shot at inference for Variational Autoencoder should be

Let's say you have an already trained Variational Autoencoder where the parameters are $\phi, \theta$ for the recognition and generative models respectively. Let's also assume you have the following ...
0
votes
1answer
54 views

Whats meant by the identity function in this question?

It is generally desirable in the context of perceptron learning to have a trainable threshold s. Prove that a one-input neuron with a fixed threshold s =−1 could ...
0
votes
0answers
32 views

Generalised Estimating Equation (GEE) vs. Recurrent Neural Network (RNN)

Has anyone looked into or know what is the difference between a GEE model and an RNN model in terms of what these two models are doing? Apart from the differences in structure of these two models ...
0
votes
1answer
59 views

Derivative for a softmax-like function

Recently, an improvement to differentiable neural computer was proposed in this paper (Improving Differentiable Neural Computers Through Memory masking, De-Allocation, and Link Distribution Sharpness ...
0
votes
0answers
15 views

Finding effect of inputs on output (shapely values)

I've developed a neural network which takes in n inputs returning m outputs. I want to see which inputs contribute most with each output. One idea I had is for all inputs/output combinations, lock ...
2
votes
0answers
49 views

What is the mathematics behind the architecture of AlexNet?

I've been trying to learn more about convolutional neural networks (coming from an SVM background) and I've been struggling with understanding how decisions were made when designing some of the ...
2
votes
0answers
53 views

Tensor chain rule: Backprop notation that is compatible with the chain rule?

The usual way backprop is written for neural networks is "inconsistent" with the chain rule. To show this, let's use a linear neural network y=ABx (for simplicity, think of A and B as $\in R^{n\times ...

1
2 3 4 5
11