63
$\begingroup$

I'm embarrassed to say that I have a PhD and hold an asst professorship, but get tripped up when reading statistics research. I am in a field of Business that is similar to IO Psychology or Social Psych. I spend too much time reading applied stats books, but I find even with all the reading I don't have a firm grasp of what I'm actually doing. Everything is very 'seat of the pants.' (As sad as it seems, I think this is not a unique situation among the faculty in the social sciences...) The biggest problem comes when I need to apply a rarely used stat technique. I can find an article from a mathematical stats journal with the equations that would solve my problem, but I don't have the math to convert those into code. I am forever relying on other prof's R packages, and crossing my fingers hoping it will work (I can't even check to verify if it did or not). It's been over 15 years since I took Calculus and Algebra in undergrad, and I think I want to start at the beginning and truly understand probability and statistics.

I am starting with Gelfand's Algebra and Trigonometry books for a quick refresher of the basics -- I know it's hard to believe, but in an applied research field we rarely have use for sin or cos. I'm even trying to finally learn how to correctly do a proof, using the books from Velleman ("How to Prove It") and Houston ("How to Think Like a Mathematician") -- I'm serious about doing this right and understanding the subject. From there I want to move on to (correctly) learn the Calculus and Linear Algebra I need to tackle probability and statistics. I was thinking of using Strang's Calculus and Algebra books. But Apostol's Caculus comes highly recommended as well. After that I am completely at a loss. Further, I don't know how far to go into Calculus or Linear Algebra before I reach diminishing returns. (Apostle introduces Probability in the second half of Vol. 2 -- is it vital that I work through everything preceding it before tackling Probability?)

So my question is: if you had to do it over again with the goal of truly, deeply understanding statistics, where would you start? What books are the modern path to deep understanding? I would like to follow a modern path so that I can understand current research in statistics, including Bayesian approaches. But not in a machine learning context (which seems to be the all the rage at the moment), rather a social science / design and analysis of experiments / multilevel modeling context. Perhaps my goal would be the work of Andrew Gelman; his and Hill's book showed me how I should be looking at modeling and statistics (simulation, uncertainty estimates everywhere, bayesian inference, and so on). How should I go about relearning this material with that end goal in mind?

Update 1: Possible texts, starting from scratch with a focus on proofs and deep understanding. Not necessarily one after another.

Relearn the basics:

Calculus (which one(s), and how deep?):

Linear Algebra (which one(s) and how deep?):

Probability (which one(s)?):

Core Statistics (which one(s)?):

Other suggestions? Again with the goal of understanding and developing (or at least implementing) new methods in hierarchical modelling (generalized and linear).

|cite|improve this question
$\endgroup$
  • 4
    $\begingroup$ Great question. Velleman's books certainly is the beginning of the path to ultimate enlightment. $\endgroup$ – Git Gud Jul 4 '13 at 22:12
  • 4
    $\begingroup$ This is a very clearly and maturely formulated question: +1 (and I'm sorry to be completely incompetent to answer it...) $\endgroup$ – Georges Elencwajg Jul 4 '13 at 22:13
  • 3
    $\begingroup$ This is very serious and mature approach, and the knowledge how correctly do a proof is essential. On the other hand, as calculus, algebra and trigonometry are great for personal enjoyment, however, those are not necessary for basic probability theory (finite/counting spaces). I think that those simple models would greatly increase your understanding, and only after that I would suggest going for reviewing calculus and linear algebra, so that you can study more advanced topics later. Finally, try some online lectures (e.g. at Khan Academy), maybe it will work for you. $\endgroup$ – dtldarek Jul 4 '13 at 23:05
  • 3
    $\begingroup$ This is a somewhat anecdotal and indirect answer, but at my university, one can take (serious, intended for math majors) first courses in probability and statistics with just first-year calculus and linear algebra as prerequisites. Do you not also get to take courses at your university for free if you are a professor? If this is the case, (and if the courses' prerequisites are similar to what I described), why not take 1 course per semester over 2 years or so to get formal training in the basics? $\endgroup$ – Omnivium Jul 4 '13 at 23:09
  • 4
    $\begingroup$ I know of profs who decide to teach a subject, in order to understand it better. Perhaps you could try this, since it forces you to know it well in order to explain it? $\endgroup$ – Calvin Lin Jul 4 '13 at 23:24
42
$\begingroup$

As someone who started out their career thinking of statistics as a messy discipline, I'd like to share my epiphany regarding the matter. For me, the insight came from Linear Algebra, so I would urge you to push in that direction.

Specifically, once you realize that the sum of squares, $\sum_i X_i^2$, and sum of products, $\sum_i X_i Y_i$, are both inner products (aka dot products), you realize that nearly all of statistics can be thought of as various operations from linear algebra.

If you sample $n$ values from a population, you have an $n$-dimensional vector. The sample mean is a projection of this vector onto the $n$-dimensional all-ones vector. The standard deviation is projection onto the $(n-1)$-dimensional hyperplane normal to the all-ones vector (finally an intuitive reason for the "$n-1$" in the denominator!). Specifically, for the sample variance $s^2$ for sample $X$, here is the linear algebra:

First, we work with deviations from the mean. The mean in linear algebra terms is

$\bar{X}=\frac{\langle X,\mathbf{1}\rangle}{\langle \mathbf{1},\mathbf{1}\rangle} \mathbf{1}$

where $\langle \cdot, \cdot \rangle$ is the inner product and $\mathbf{1}$ is the $n$-dimensional ones vector. Then the deviation from the mean is

$x = X - \bar{X}$

Note that $x$ is constrained to an $(n-1)$-dimensional subspace. The usual equation for variance is

$s^2 = \dfrac{\sum_i (X_i - \bar{X})^2}{n-1}$

For us, that's

$s^2 = \dfrac{\langle x, x \rangle}{\langle \mathbf{1}, \mathbf{1} \rangle}$

which, without going into too much detail (too late) is a normalized deviation. The trick there is that the new $\mathbf{1}$ has dimension $n-1$.

The other good example is that correlation between two samples is related to the angle between them in that $n$-dimensional space. To see this, consider that the angle between two vectors $v$ and $w$ is:

$\theta = \arccos \dfrac{\langle v, w \rangle}{\|v\|\|w\|}$

where $\|\cdot\|$ is vector length. Compare this to one of the forms for the Pearson Correlation and you will see that $r = \cos \theta$.

There are many other examples, and these have barely been explained here, but I just hope to give an impression of how you can think in these terms.

|cite|improve this answer
$\endgroup$
  • 4
    $\begingroup$ I never thought of it that way. This is the kind of understanding I'm hoping for. Is there more, and a source for this inspiration? Or is it just something we need to figure out on the way? It does help explain why all the higher level GLMM books (Stroup's recent one, for ex) basically treat modelling as a question of making sure you put your data into the correct matrices. $\endgroup$ – user27634 Jul 11 '13 at 3:09
  • 2
    $\begingroup$ For me, it took a stats professor to make an offhand comment about the correlation essentially being the angle between the two vectors, then it clicked with the linear algebra I was familiar with. I don't have any sources that talk about this (though they may exist). I'll update my answer with a bit more detail though. $\endgroup$ – Nigel Jul 11 '13 at 5:46
  • 1
    $\begingroup$ I found this book recently, although I'm only part of the way through it and couldn't really recommend it. Might be useful to look at though. (Statistical Methods: The Geometric Approach amazon.com/gp/product/0387975179/…) $\endgroup$ – hadsed Nov 29 '13 at 23:39
  • $\begingroup$ I do not have a background in statistics, but I am very much interested in the geometry underlying statistics. I started out with the book @hadsed mentioned, but I now prefer Wickens' The Geometry of Multivariate Statistics. $\endgroup$ – Michael Wijaya Dec 25 '15 at 21:47
6
$\begingroup$

My humble contribution to your book list: Linear Algebra Done Right by Axler. It's a brilliant book that makes a lot of abstract things very clear. It had been recommended to me many times.

Also, I recently found a book entitled Statistical Methods: The Geometric Approach. I haven't read through all of it yet, but it gives a very basic introduction to probability from a linear algebra perspective, which I think is very intuitive (much easier on the eyes than looking at sigmas with a bunch of random indices I feel).

(Sorry, I'm too noob on this website to post a comment.)

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Incredible book. The brilliance of Axler's text is how he tiptoes around using the determinant until the end of the book. Not to say that the determinant is not important; on the contrary, it is one of the most powerful L.A. tools. However, Axler provides you a solid understanding of vector spaces, operators, eigenvalues, decompositions, the characteristic polynomial, BEFORE exposing you to the trace/determinant. $\endgroup$ – A.E Jul 10 '13 at 23:42
  • 4
    $\begingroup$ As a counter-opinion, I despise this text. The material covered is fine, but it's presentation is hectic and contradictory. He finds it necessary to expound on useful concepts, but then leaves more advanced notions almost completely undefined. He presents definitions, then fails to support those definitions with useful exercises of why they are important or relevant to the concept at hand. To read this text, you need to accept prima facie everything he says, while simultaneously trying to embrace the mathematical notion of exploring the meaning of everything. $\endgroup$ – Emily Jul 10 '13 at 23:55
  • $\begingroup$ Also, the professor I had who used that book was awful. Simply awful. $\endgroup$ – Emily Jul 10 '13 at 23:55
  • $\begingroup$ I was just about to suggest this book. I don't recall seeing the problems referenced by Arkamis, but perhaps I had a good professor who covered these things up. $\endgroup$ – Nigel Jul 11 '13 at 0:02
  • $\begingroup$ @Arkamis, it makes me sad that you had such a bad experience with the book. Given that you had such a seemingly terrible professor, I highly recommend skimming parts of the book again to see if you still feel that way. To be honest, even in my own readings of the book w/o instruction (I continued my studies where the class left off in Ch8), I have not encountered the frustrations you mention. $\endgroup$ – A.E Jul 11 '13 at 0:49
2
$\begingroup$

I think to 'truly, deeply understand' statistics, you have to understand probability theory . Here's some resources to gain a strong conceptual foundation:

Harvard Stat 110 http://projects.iq.harvard.edu/stat110 The psets are gold.

The MIT course on Applied probability is of equal quality, and you can find it on edX

https://www.edx.org/course/mitx/mitx-6-041x-introduction-probability-1296

An informative and entertaining read to hone intuition:

http://www.amazon.com/Lady-Luck-Theory-Probability-Mathematics/dp/0486243427

|cite|improve this answer
$\endgroup$
0
$\begingroup$

Thank you for the the question and the background story which resonates with my experience in social science. My current plan is:

  1. Set theory and logic
  2. Abstract algebra
  3. Linear algebra
  4. Topology
  5. Calculus and analysis
  6. Measure theory
  7. Probability theory

I wonder whether to invest time in information theory and inverse problems.

Several years have passed after posting the original question. What has your experience been like?

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.