13
$\begingroup$

Most of the literature I can find in the field of machine learning is extremely practical, listing many techniques you can use like neural networks, SVMs, random forests, and so on. There are lots of suggestions on implementations and what approaches are good for different problems. My question is, is there a single unifying formalism or theoretical framework in which one can think about problems of inferring patterns and conclusions from data?

In such a framework I'd be interested in exploring these kinds of questions:

What is the correct number of free parameters to use for a given data set? How do we know when we've over-fit? How do we express the trade-off between the accuracy of low-resolution fitting and the precision of high-resolution fitting?

Can we define a machine learning techniques in general terms (instead of specific ones like neurons or forests or deep learning or graphical models or...)? How can we objectively compare these techniques? Are any of them actually identical?

What kinds of signals are possible to extract from data? What kinds of noise could prevent successful learning?

$\endgroup$
  • 3
    $\begingroup$ I doubt there is a particularly good answer to your question: the experts in machine learning are much more focused on finding practical solutions to real problems than formulating an elegant theory. However there do seem a few unifying themes: convex optimization in regression algorithms, entropy in classifiers, and the Johnson-Lindenstrauss lemma in sparse matrix theory and dimension reduction. $\endgroup$ – Paul Siegel Mar 20 '14 at 19:57
  • 2
    $\begingroup$ Maybe you're looking for a computational learning theory, such as PAC learning or VC theory. $\endgroup$ – Rahul Jan 4 '15 at 19:39
1
$\begingroup$

I agree with the other respondent. The focus on machine learning is more geared towards ready-to-implement algorithms. However there are some Simple results like the Bias-Variance Tradeoff and other sort of discussions of "limiting" performance. From what I've seen, your question might be most practically addressed by the area of model selection (i.e. how to pick one of the algorithms you mentioned to solve a problem). In some cases, there is no single best algorithm, other than to use both (or several ) algorithms in an ensemble method, e.g. boosting, which takes advantage of when different learning algorithms are best at classifying different types of examples.

$\endgroup$
1
$\begingroup$

One of the general formalism for using ensemble methods for classification was developed by this author and his followers https://en.m.wikipedia.org/wiki/Yuri_Zhuravlev and called algebraic theory of algorithms, but i haven't seen any book or course in english, may be some papers, most information in russian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.