Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

0
votes
0answers
37 views

Can real vector space be a model of first order logic? [on hold]

As far as I understand the linear vector algebra is first order theory, some instance of first order logic. But what about generality? Can ve construct real (or trascendental, hypercomplex or polyadic)...
0
votes
0answers
10 views

Follow error propagation in a neural network

I am currently trying to study the impact of approximate training of an encrypted neural network. For this, my weights are all encrypted with a fully Homomorphic encryption scheme. For my purpose, ...
0
votes
0answers
14 views

What does it mean for a Wavelet transform to “commute” with a translation?

I'm referencing this paper here: https://arxiv.org/pdf/1203.1513.pdf Within this paper, it states that "A wavelet transform commutes with translations, and is therefore not translation invariant". ...
1
vote
1answer
42 views

Policy gradient reinforcement learning for continuous state and action space

I am a novice in the field of machine learning, I have a moderate level understanding of linear/non-linear regression, support vector machines, neural networks, and q-learning (for discrete finite ...
1
vote
0answers
31 views

Variable transformation for training a machine learning model

Suppose you have a train set $\mathbf{T}$ and you want to train some Machine Learning models. Each row of $\mathbf{T}$ consists in a set(vector) of attributes or variables $\mathbf{x} = (x_1, x_2...)$ ...
0
votes
0answers
22 views

Variance of normal distribution and binary vector

Note: this question is related to the maths of Neural Nets, if you need clarification about the question do comment. Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates ...
0
votes
0answers
15 views

Gradient optimization with attractors/preferences, that respects already emerged structures in parameter subspace?

One can assume that certain structures have emerged in the space of parameters which are being optimized to achieve some minimum of some function over those parameters, e.g. f(p1, p2, ...). By ...
0
votes
1answer
24 views

Why does derivative of sum(u) result in vector of 1s?

I came across these formulas (please see attached) in a paper. Could you please explain why the derivative of sum(u) where u is the dot product of vectors w and x results in vector of 1s? Thanks ...
0
votes
1answer
40 views

Functions of vectors and vectors of functions

In page 22 of the Matrix Calculus For Deep Learning, the authors wrote: It is the nature of neural networks that the associated mathematics deals with functions of vectors not vectors of functions. ...
0
votes
0answers
10 views

Problem on finding the set of biases and weights in a specific neural network

I have a doubt regarding an exercise here. Suppose that we have a neural network that tries to map a $28\times 28$ image of a digit to what digit it actually represents. So we have a neural network ...
3
votes
0answers
31 views

Gradients of Deep Neural Network With Batch Normalization

How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation) \begin{align} L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=...
1
vote
1answer
27 views

Complexity of first, second and zero order optimization

I am currently reading Bishop - 'Pattern Recognition and Machine Learning' (2006) where he writes about why using gradient information for optimization is superior to not using it. (p. 239) ...
1
vote
0answers
45 views

Directional Derivatives and Jacobian of a Linear Neural Network

I have to compute the following double derivative: $$ \partial _{x_i} \nabla_W \sigma(f(W,x))$$ where $W = (W_1, W_2, \dots, W_L)$ is the set of weight matrices, $f(W,x)$ is a $linear$ neural ...
1
vote
1answer
37 views

Graph valued function, mapping from reals to graph (maybe about realization of functors)?

Are there mappings from vectors/matrices of reals (or vectors/matrices of real-valued functions) to the graph? Motivating example: network of synapses and neurons as the domain and knowledge graph as ...
1
vote
0answers
43 views

Can we generalize the gradient computation for Deep Neural Network for more than 2 hidden layers?

In this post, thanks to greg, a solution for the gradient computations for the 2 hidden layers (or 3 layers in total per se) is presented. Now, if we want to generalize for the $L$ layers, then this ...
3
votes
1answer
85 views

Why are eigenvectors important for Deep Learning applications?

I know it is quite of a trite question to ask about the importance of eigenvectors, but I do not understand how they can be relevant for Deep Learning and when we can use them. Any reference to the ...
0
votes
0answers
25 views

True accuracy of neural network

My goal is to calculate the probability to correctly classify an object if I make $k$ predictions on slightly different images of it. The predicted class would then be the one that was predicted the ...
0
votes
1answer
21 views

Question about the notation used in the famous UFLDL tutorial on neural networks

I am trying to understand the notation used in this famous tutorial http://ufldl.stanford.edu/wiki/index.php/Neural_Networks On the very first line, the $i$ in $(x^{(i)},y^{(i)})$ is used to ...
11
votes
2answers
172 views

Why do deep neural networks work well?

The universal approximation theorem, as I understand it, states that for any continuous bounded function $f: X \rightarrow \mathbb{R}$ with compact domain $X$ and any threshold $\varepsilon$ there is ...
19
votes
5answers
1k views

What are the best books to study Neural Networks from a purely mathematical perspective?

I am looking for a book that goes through the mathematical aspects of neural networks, from simple forward passage of multilayer perceptron in matrix form or differentiation of activation functions, ...
0
votes
0answers
13 views

Proving the learnability of XOR function by a particular neural network

Let's say I have the following neural network and the constraints: The architecture is fixed (see the network in this image, I'm not allowed to post images due to low rep) (note that there are no ...
1
vote
0answers
26 views

Rewrite an one-layer neural network into infinite-layer equivalent network

Problem In the paper I am reading now, the author wrote generalized linear predictor with ReLU activation $$\{\mathbf{x}\mapsto \sigma(\mathbf{w}^T\mathbf{x}): \Vert\mathbf{w}\Vert_2\leq M\}$$ into a ...
1
vote
0answers
58 views

Derivative of Binary Cross Entropy is always negative

I'm trying to find derivatives of a back propogation algorithm. Given a loss function $$L(\hat y_i, y_i) = \sum - y_i log (\hat y_i)$$ where $\hat y_i = \sigma(z)$ and $z = Wx + b$. Find the update ...
0
votes
1answer
69 views

Finding a global minimum

I seek the function $f$ which satisfies the 100 equations (i=1,2...100) $\sum_{j=1}^{2000} f(A_{ij},B_{ij},C_{ij})=Q_i$. Where $A,B,C$ are 100x2000 matrices and all entries are between 0 and 1. ...
4
votes
0answers
63 views

Algebra & Artificial Intelligence (AI)

Artificial intelligence, especially deep learning & neural networks for image processing and classfication, are related to statistics and physics e.g. as decribed in below papers. Statistics and ...
0
votes
0answers
19 views

Classical gradient descent optimization of smooth nonnegative function $f$ restricted to hypercube $C = [-R,R]^s$.

Let $f : \mathbb{R}^s \rightarrow \mathbb{R}_{\geq 0}$ be a smooth, nonnegative function and $R > 0$. Now chose a point $x_0 \in (-R,R)^s$ in the interior of the compact hypercube $C := [-R,R]^s$. ...
1
vote
0answers
43 views

Directional Derivative of Softmax

I'm trying to solve an exercise about computing derivatives with softmax, but I'm somehow stuck. I have a deep neural network $f(W,x)$, where $W$ are the weights, and I have a fixed weight-vector $\...
2
votes
1answer
158 views

Category Theory & Artificial Intelligence (AI)

Category theory turns out to be useful in more and more areas. (see e.g. MSE - Category Theory & Biology) Question. Does anyeone know of some connection of category theory to (convolutional) ...
2
votes
0answers
39 views

Is there such branch as Nonlinear Matrix Algebra?

Is there such branch as nonlinear matrix algebra, that researches nonlinear functions with matrix (tensor) valued arguments and outputs where nonlinearities are applied componentwise. Such functions ...
2
votes
1answer
24 views

Universal Approximation with Fixed Layer

Fix an activation function $\sigma$, and denote the class of all Neural-networks from $\mathbb{R}\rightarrow\mathbb{R}$ defined by this activation function by $NN^{\sigma}$. The classical universal ...
1
vote
0answers
39 views

Derivation of partial derivative of cost function with respect to weights in backpropagation algorithm

I am studying Machine Learning from Andrew Ng's Machine Learning course on coursera. I am stuck at understanding math behind back propagation. Here is an image of backpropagation algorithm from his ...
1
vote
1answer
25 views

Suggestion of article or book for convolution

I need a good book or article to learn about convolution.I have a course in Neural Networks and we have to make calculations by hand.
5
votes
0answers
71 views

Machine learning book with robust linear algebra approach

I am looking for machine learning book - neural network, deep learning etc etc - that use linear algebra in a robust manner. I found satisfactory the old book of Simon Haykin : Neural Networks : A ...
0
votes
1answer
23 views

Changing signs of partial derivative's elements

If $C = \frac{1}{2}(y - a)^2$ where $y$ is a given value, $a = \sigma(z)$, and $z = wx + b$. Then the partial derivative of $C$ with respect to w should be: $\frac{\partial C}{\partial ...
1
vote
0answers
53 views

Deriving the error in activation nodes in back propagation algorithm

I am trying to understand back propagation algorithm from Andrew Ng's Machine learning course. Here is a pitcure of the slide on which I am stuck. I know that error in a function $f(x)$ is calculated ...
3
votes
1answer
57 views

Are neural networks with bounded parameters a compact subset of the Banach space of continuous functions?

Let $d, n \in \mathbb{N}$. Moreover, let $D \subset \mathbb{R}^d$ be compact and denote with $\mathcal{C}(D, \mathbb{R}^n) $ the set of continuous functions from $D$ to $\mathbb{R}^n$. Then $\mathcal{...
1
vote
1answer
33 views

What is the output in a RNN?

I have recently been looking for some information about recurrent neural networks. Some people use a layer between the hidden state and the output and other ones use the hidden state as output. What ...
1
vote
1answer
26 views

Proof Related to Convolutional Neural Network

I would to know why Convolutional Neural Network(CNN) works. It is known from Universal Approximation Theorem that a feedfoward neural network with a single layer can approximate continuous functions. ...
0
votes
0answers
39 views

What's going on in this sum?

This is part of a slope calculation example to update the new weight of a neural network from the slope. From a Deep Learning course on DataCamp My math is a bit dodgy and what I don't understand is ...
0
votes
1answer
35 views

How to update the weights knowing the loss in Neural Network

Question How update the weight knowing the loss Work explanation I have a really simple network composed by two layers with one neurons in both layers. Considering an input of 1, the final result ...
1
vote
0answers
24 views

Reversible analog coding of strings (mathematical expressions)

Neural coding converts single word or single sentence of words into the vector of real numbers. This coding, while sometimes useful, is not revertible. Are there methods (or at least research effort ...
0
votes
0answers
13 views

Distance between neural networks and symmetry thereof

As artificial intelligence has gotten fancier and fancier these last few years, I would like to explore a little the issue of the structure of neural networks. First I would like to know if there is a ...
0
votes
0answers
17 views

Function multiplication in the frequency domain

I need to solve the following problem $$ \int_{-U}^{U} f(x,t) k(x) dx = H(t) $$ where $f(x,t)$ is the transition density of a random variable $X$ often a gaussian, i.e.: $$f(x,t) = \frac{1}{\...
0
votes
0answers
9 views

“Univariate function” definition for functions of vectors?

As I understand it, the definition of a univariate function is a function of a single variable. However, I am reading through some texts on machine learning and neural networks, and they frequently ...
2
votes
1answer
65 views

Introductory mathematical books / courses to take before reading Ian Goodfellow: Deep learning

I only have a German high school education in mathematics. What mathematical courses/books/concepts would you recommend me to study before reading "deep learning" by Ian Goodfellow? My high school ...
0
votes
1answer
17 views

Conserving variance of normally disributed vector in linear transformation

I have normally distibuted vector $X$ with known variance $Var(X)$ and mean $\overline X$ and then I take new vector $Y$ as $W*X$ where $W$ is normally distributed matrix. If I required $Y$ to have ...
0
votes
0answers
26 views

Given an input image of 128x128, and output 29x29. What is the size of the area of the input that is middle pix in output

Given an input image of 128x128, after putting it through a convolutional neural network the output is 29x29. What is the size of the area (in pixels) of the input that correspond to the middle pixel ...
0
votes
0answers
24 views

Radial Basis Function

I have this question and I need help; What is the effect of using a more hidden layer on the performance of approximation of the RBF network?
0
votes
1answer
13 views

Radial Basis Function Network

I have a problem in this code: Error using plot Vectors must be the same length. plot(t,ge); ...
0
votes
1answer
41 views

Values in Softmax Derivative

I am trying to correctly understand the derivative of the softmax-function so that I can implement it correctly. I already know that the derived formula looks like tbis: $\frac{\delta p_i}{\delta a_j}...