# Questions tagged [neural-networks]

For questions about the mathematics of artificial neural networks: their underlying multilayered graph object or their use as a data structure in machine learning algorithms. Consider also using the tags (machine-learning) or (graph-theory).

622 questions
Filter by
Sorted by
Tagged with
9 views

### The maximal singular value or condition number of such random matrix product.

Let ${X}_i\in\mathbb{R}^{d\times d}$ denote $d$-by-$d$ random matrix. Every element of ${X}_i$ is sampled from Gaussian distribution $\mathcal{N}(0, 1/d)$ independently. Define ${R}_n$ as ($I_d$ is ...
9 views

### What scaling is appropriate for prediction intervals

I have a set of price forecasts which I am feeding in to a neural network for another prediction. Included in this price forecast are prediction intervals at 95, and 50% levels. What is the best way ...
14 views

### Details of proof of convergence of SGD

Deriving the SGD rule I came across some essential doubts about the stochastic gradient. In the proof that I'm reading, they introduce the subgradient $g$ together with the iteration rule as follows ...
55 views

### Variance of $\tanh z$ where $z\sim{\cal N}(0,\sigma^2)$

In Priors for Infinite Networks (Neal, 1996), the paper considers a simple one hidden layer neural network defined by \begin{align}h &= \tanh (a + Ux) \\ f &= b + Vh\end{align} where $f(x)$ is ...
17 views

22 views

### Formal robustness verification of neural networks: MILP vs SMT

I'm not sure if this is the right place to ask this questions. I'm working my way into the field of formal verification of neural networks. The goal is to analytically evaluate the robustness of ...
63 views

### Calculating derivatives with respect to a weight matrix

I am currently doing a machine learning course and I am trying to wrap my head around back propagation. I watched these videos which helped clear things up a bit. I am trying to apply the same ...
63 views

### Neural network architecture capable of performing a sum by category?

I am wondering whether it would be possible to build a NN that can be trained to take 2D training examples (with a fixed number of rows) where the two columns would represent an amount and a category, ...
14 views

### Gradient norm in a neural network is bounded?

Consider a fully connected neural network with single hidden layer $f(x,w) = w^T_2 \sigma(w^T_1 x)$ where $w = [ w_2, w_1 ]$ are networks' parameters and $\sigma$ is an activation function (e.g tanh, ...
31 views

### Context: “Squashing Functions and Neural Networks”

I hope your lives have been proceeding along well. Lately, I have been reading about squashing functions in the context of neural networks. Specifically, the book I am working through, Deep Learning ...
31 views

19 views

### Partial-derivative in an artificial neural network. Product of vectors of different length?

Introduction I am programming an artificial neural network to analyze the MNIST dataset of handwritten digits. Vector $\textbf{a}$ in layer $\textit{L}$ of length $\textit{i}$ in the network are given ...
35 views

### Problems in getting the derivatives of the batch normalization layer

I'm working on understanding the math used in the batch normalization layer in the CNN and found the original paper discussing this trick: Batch Normalization: Accelerating Deep Network Training by ...
110 views

### Are Resnet Neural Networks non-affine transformations with linear activation functions

Given an Artificial Neural Network that allows "skip" connections or inter-layer connections such as found in Resnet, is it possible to get non-affine mappings of the input to the output ...
52 views

### How can the $\tanh(x)$ activation function be written for two variables? $\tanh(x,y)$

The tanh activation function is: $$\tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Is there a corresponding formula for variable pairs? $$\tanh(x,y) =?$$ My uneducated guess is that ...
21 views

### Accurate metric around zero for neural networks

I´m trying to approximate a continuous function via regression with a neural network. The function takes values in the interval $[-10, 10]$, but the values around zero are more important, because, in ...
28 views

### Dirichlet Energy for Graphs, Derivation

I would like to prove this formulation of the Dirichlet Energy for Graph Neural Networks  \begin{aligned} E(\mathbf{X}) &=\frac{1}{d_{i}} \sum_{j \in \mathcal{N}(i)} w_{i j}\left\|\mathbf{x}_{i}-...
I'm trying to solve one optimization using a neural network. Let's say, I have a neural network, i.e., a function $f$ operating on a matrix and outputs another matrix. Two things are given: $\nabla f$...