Cascade Neural Networks Recently I wondered if there is a neural network topology that can be flexible and adapt to new kinds of data and able to be trained continuously.I found a good paper on Cascade Neural Networks and I think this topology overcomes many of the problems that traditional NN's suffer such as choosing right number of layers or hidden units.  
Reading the paper it sounds as if this is the perfect architecture - quick learning , high accuracy , adaptive to changes.How come this topology isn't as widely spread as deep networks, which are so hard to fine tune and train?
I can't find any real world problems that were solved with Cascade NN.What are the cons of using this architecture ?
 A: One disadvantage is that it is much harder to implement than a standard multilayer Perceptron. Another disadvantage is that this is for "standard" feed forward networks, but not for CNNs / RNNs.
One architecture which is closely related to the cascade part and is for CNNs are the recently developed Dense Nets: https://arxiv.org/abs/1608.06993
I am pretty sure DenseNets will be wide-spread quite soon.
A: There's a neat question in here (though the more interesting part is buried in comments), and several relevant topics that people interested in this might like (even four years later).

How come this topology isn't as widely spread as deep networks, which are so hard to fine tune and train?
I can't find any real world problems that were solved with Cascade NN. What are the cons of using this architecture ?

Well as you say,

I think this topology overcomes many of the problems that traditional NN's suffer such as choosing right number of layers or hidden units.

but what is the cost of this generality? One of the main benefits of deep architectures is the ability to do end-to-end training and gradient calculations. But this algorithm cannot do so. As noted in the paper:

We add hidden units to the network one by one.  ... The hidden unit's input weights are frozen at the time the unit is added to
the net; ...

In general, any kind of architecture search is non-differentiable. This is why fixed architectures are so popular, and searched via e.g. cross-validation, instead of during the training of a single model.
I suspect this kind of model would be very difficult to train to the same level of accuracy as a large deep network, even if it were easy to implement, since the training is not really optimizing the entire network construction procedure at once (or even the whole network at any fixed point in time).
Your comments:

it still seems quite useful for constantly changing data flow.I don't like the notion of 'neural network' for that reason - it's static.People learn in real time all the time and adapt.
What I originally wanted to know was exactly this, if there's a neural net that is trained continuously and not with a static dataset.

It seems you'd be more interested in the kind of problems where not only is the network getting continuous data (known as online learning], but that data is changing (formally, the domain of the training data is changing) and/or the task is changing. It might be worth it to look at

*

*Domain adaptation, which looks at how to generalize neural networks trained on one dataset $D_1$ to another dataset $D_2$, very similar to transfer learning. For instance, given a network able to detect cats, how can I use it to detect dogs as well, without starting from scratch?


*Active learning, which considers when the learner is involved in the data selection process itself (i.e., it can decide which data points are most useful to its learning, depending on the task).


*Lifelong learning (also called continuous learning) deals with the more generous case of both non-stationary data and dynamic tasks. It's closely associated with the well-known problem of catastrophic forgetting in neural networks, when tackling the sequential multi-task case.


*Neural architecture search, which looks at how to optimize the complex hyper-parameters controlling the neural architecture (layer number and type, etc...).
