3
$\begingroup$

Setting

One version of the Kolmogorov-Arnold representation theorem (cf. [1, Thm. 1]) states

Theorem: Fix $d \geq 2$. There are real numbers $a, b_p, c_q$ and a continuous and monotone function $\psi: \mathbb{R} \rightarrow \mathbb{R}$ such that for any continuous function $f: [0, 1]^d \rightarrow \mathbb{R}$, there exists a continuous function $g: \mathbb{R} \rightarrow \mathbb{R}$ with $$ f(x_1, \ldots, x_d) = \sum_{q = 0}^{2d} g\bigg( \sum_{p = 1}^{d} b_p \psi(x_p + qa) + c_q \bigg). \tag{1} $$

Definition: Two-hidden-layer feedforward neural networks with activation functions $\sigma_1$ and $\sigma_2$ (in the second resp. the first layer) are defined as

$$ f(\mathbf{x}) = \sum_{q = 1}^{m_1} d_q\, \sigma_1\bigg( \sum_{p = 1}^{m_2} b_{pq}\, \sigma_2(\mathbf{w}_p^T\mathbf{x} + a_p) + c_q \bigg) \tag{2} $$ where $\mathbf{w}_p \in \mathbb{R}^d$, $a_p, b_{pq}, c_q, d_q \in \mathbb{R}$ are parameters.

Question

According to [1], the RHS of (1) can be represented via a two-hidden-layer feedforward neural network with activation functions $g$ and $\psi$ (in the second resp. the first layer). I agree that it looks similar to (2) but I could not find a sensible choice of the parameters.

Choosing $m_1 = 2d + 1, m_2 = d, d_q = 1, \sigma_1 = g, \sigma_2 = \psi$, $b_{pq} = b_p$ where $b_p$ are independent of $q$ and $\mathbf{w}_p$ being the $p$-th unit vector, then I get $$ f(\mathbf{x}) = \sum_{q = 1}^{2d + 1} g\bigg( \sum_{p = 1}^{d} b_{p} \psi(x_p + a_p) + c_q \bigg). \tag{3} $$ However, I am stuck now because I don't understand why $a_p$ can be chosen dependent on $q$ (which choosing $a_p = qa$ would do). Is that allowed in a neural network setting?

References

[1] https://arxiv.org/abs/2007.15884 (published in Neural Networks 137 (2021) 119–126)

$\endgroup$

2 Answers 2

2
$\begingroup$

I don’t think it’s really a neural network in the traditional sense because the q appears in the place where the p layer is, which shouldn’t have any info about q. By the way, q is a constant. Here’s what it might look like.

enter image description here

$\endgroup$
2
$\begingroup$

I have found the answer in [2] which is the original reference of the above mentioned KA-representation. But I believe that the statement from 1 is nevertheless incorrect. The representation can be achieved by two-hidden-layer neural network but it needs to have more units in the first layer. Then, it would look like this picture I found in [2].

enter image description here

References

[2] https://bonndoc.ulb.uni-bonn.de/xmlui/handle/20.500.11811/4169

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .