2
$\begingroup$

Setting

One version of the Kolmogorov-Arnold representation theorem (cf. [1, Thm. 1]) states

Theorem: Fix $d \geq 2$. There are real numbers $a, b_p, c_q$ and a continuous and monotone function $\psi: \mathbb{R} \rightarrow \mathbb{R}$ such that for any continuous function $f: [0, 1]^d \rightarrow \mathbb{R}$, there exists a continuous function $g: \mathbb{R} \rightarrow \mathbb{R}$ with $$ f(x_1, \ldots, x_d) = \sum_{q = 0}^{2d} g\bigg( \sum_{p = 1}^{d} b_p \psi(x_p + qa) + c_q \bigg). \tag{1} $$

Definition: Two-hidden-layer feedforward neural networks with activation functions $\sigma_1$ and $\sigma_2$ (in the second resp. the first layer) are defined as

$$ f(\mathbf{x}) = \sum_{q = 1}^{m_1} d_q\, \sigma_1\bigg( \sum_{p = 1}^{m_2} b_{pq}\, \sigma_2(\mathbf{w}_p^T\mathbf{x} + a_p) + c_q \bigg) \tag{2} $$ where $\mathbf{w}_p \in \mathbb{R}^d$, $a_p, b_{pq}, c_q, d_q \in \mathbb{R}$ are parameters.

Question

According to [1], the RHS of (1) can be represented via a two-hidden-layer feedforward neural network with activation functions $g$ and $\psi$ (in the second resp. the first layer). I agree that it looks similar to (2) but I could not find a sensible choice of the parameters.

Choosing $m_1 = 2d + 1, m_2 = d, d_q = 1, \sigma_1 = g, \sigma_2 = \psi$, $b_{pq} = b_p$ where $b_p$ are independent of $q$ and $\mathbf{w}_p$ being the $p$-th unit vector, then I get $$ f(\mathbf{x}) = \sum_{q = 1}^{2d + 1} g\bigg( \sum_{p = 1}^{d} b_{p} \psi(x_p + a_p) + c_q \bigg). \tag{3} $$ However, I am stuck now because I don't understand why $a_p$ can be chosen dependent on $q$ (which choosing $a_p = qa$ would do). Is that allowed in a neural network setting?

References

[1] https://arxiv.org/abs/2007.15884 (published in Neural Networks 137 (2021) 119–126)

$\endgroup$

2 Answers 2

1
$\begingroup$

I don’t think it’s really a neural network in the traditional sense because the q appears in the place where the p layer is, which shouldn’t have any info about q. By the way, q is a constant. Here’s what it might look like.

enter image description here

$\endgroup$
0
$\begingroup$

I have found the answer in [2] which is the original reference of the above mentioned KA-representation. But I believe that the statement from 1 is nevertheless incorrect. The representation can be achieved by two-hidden-layer neural network but it needs to have more units in the first layer. Then, it would look like this picture I found in [2].

enter image description here

References

[2] https://bonndoc.ulb.uni-bonn.de/xmlui/handle/20.500.11811/4169

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.