How can I "see" that calculus works for multidimensional problems?

Question

Let's say I have some function f(x) = x^2 + b. I can see what's going on, I can count the slope geometrically even without knowing the rules of derivatives.

When I need to minimize some cost function for linear problem (linear regression with gradient descent), I just need to picture 3d, and I can "see", and be quite confident why and how it works.

How can I "see" or get intuition that calculus works for multidimensional problems? Let's say I have a problem with many variables like:

f(area_km, bedrooms, ...) = theta + theta1 * area_km + theta2 * bedrooms etc

If I want to apply gradient descent, I know I need to calculate partial derivatives and multiply it with a learning rate etc. It works. But it's kinda magical that it works.

I am sorry this is a silly question, I am just beginning.

Answer 1

For the most general case, think about a mixing board.

Each input argument to the function is represented by a slider with an associated piece of a real number line along one side, just like in the picture. If you are thinking of a function which can accept arbitrary real number inputs, the slider will have to be infinitely long, of course, which of course is not possible in real life, but is in the imaginary, ideal world of mathematics. This mixing board also has a dial on it, which displays the number corresponding to the function's output.

The partial derivative of the function with respect to one of its input arguments corresponds to how sensitive the readout on the dial is if you wiggle the slider representing that argument just a little bit around wherever it's currently set - that is, how much more or less dramatic the changes in what is shown are compared to the size of your wiggle. If you wiggle a slider by, say, 0.0001, and the value changes by a factor 0.0002, the partial derivative with respect to that variable at the given setting is (likely only approximately) 2. If the value changes in an opposite sense, i.e. goes down when you move the slider up, the derivative is negative.

The gradient, then, is the ordered list of signed proportions by which you have to "wiggle" all the sliders so as to achieve the strongest possible, but still small, positive wiggle in the value on the dial. This is a vector, because you can think of vectors as ordered lists of quantities for which we can subject to elementwise addition and elementwise multiplication by a single number.

And of course, when I say "small" here I mean "ideally small" - i.e. "just on the cusp of being zero" which, of course, you can make formally rigorous in a number of ways, such as by using limits.

Answer 2

Well, in one variable you need to solve $f'(x)=0$ and controll wheter it is maximum or a minimum. Such equation is called Euler equation and it holds also in more variables, but with the formulation: $$\nabla f(x)=0$$ where $\nabla f(x)=(\frac{\partial f(x)}{\partial x_{1}},\dots,\frac{\partial f(x)}{\partial x_{n}})$. Now, also here you should controll if it is a minimum or not and this is done via checking on the Hessian of $f$. That is, if $\bar{x}$ is such that $\nabla f(\bar{x})=0$ then $\bar{x}$ is a minimum if $Hf(\bar{x})$ admits only positive eigenvalues, where $Hf$ is the matrix made of second order partial derivatives of $f$. Notice that we are assuming $\bar{x}$ belongs to the interior of a set, exactly as for functions with one variable.

Answer 3

Consider a single variable function $f(x)$, suppose there exists a maximum for $f(x)$ , then we can find that by the function's derivative has a sign flip at that point, then if we were to take a point $ x>x_o$

$$ \frac{df}{dx}|_{x>x_o} = \text{something negative}$$

Now, the magnitude of the derivative depends on how far you are from the global maximum, so suppose you take a 'step' on the $x$ axis scaled up by the derivative.

$$ \frac{df}{dx}_{x > x_o} \Delta x$$

Then, you will end up walking to the maximum. Now let's say you are at a point $x<x_o$ , then the first derivative is positive and you will still end up walking toward the maximum. Moral of the story? If you walk around the input set keeping your steps scaled up by the function's derivative, then you'll eventually hit a global maximum / minimum.[Edit: It may also turm up that you get stuck in local mini/global minimum :(]

Now, consider a multivariable $f(x,y)$ , by the logic above if it has a local max, say at a point $(x_o,y_o)$, if you take a $x>x_o$, then

$$ \frac{\partial f}{\partial x} = \text{something negative}$$

And similar argument to single variable case can be applied, and we can apply a similar argument for $y$. Ultimate this leads us to idea that the vector given as:

$$ \nabla F = < \frac{\partial F}{\partial x} , \frac{\partial F}{\partial y} >$$

Tells us how to move in the input plane such that our function is maximized.

So, say you are a point $<x_o,y_o>$ , then the point where you should move next to maximize the function is:

$$ <x,y> = <x_o,y_o> + < \frac{\partial F}{\partial x}|_{x_o} \Delta x, \frac{\partial F}{\partial y}|_{y_o} \Delta y>$$

Why? If $z(x,y)$

$$\Delta z= \frac{\partial F}{\partial x} \Delta x + \frac{\partial F}{\partial y} \Delta y= \nabla F \cdot ds$$

$ds$ is the length of step you take in the input plain, clearly for a fixed step length, the most increase in function happens when the angle between step and gradient is zero.

Hence, using that gradient vector as a compass to move, you'll finally reach some kind of extremum point in the input plane.

Answer 4

$\def\p{\partial} \def\vr{{\bf r}} \def\th{\theta} \newcommand\pder[2][]{\frac{\partial #1}{\partial #2}} \newcommand\der[2][]{\frac{d #1}{d #2}}$Nontrivial geometrical intuition can be obtained by considering temperature as a function of position in three dimensions. Suppose temperature is given by $T(\vr)$, where $\vr = \langle x,y,z\rangle$. Surfaces of constant temperature ("equipotentials") are given by $T(\vr) = \mathrm{const}$. We wish to minimize the temperature starting from $\vr=\vr_0$ where $T(\vr_0)=T_0$, that is, we wish to move from this equipotential to another with lower temperature following the shortest possible path. We will argue that we must move in a direction orthogonal to the equipotential. In fact, we must move in the direction opposite that of the gradient of temperature.

Consider a path through $\vr_0$ parametrized by $t$. By multivariable chain rule, the rate of change of $T$ along the path is given by \begin{align*} \frac{d T(\vr(t))}{dt} = \pder[T]{x}\der[x]{t} +\pder[T]{y}\der[y]{t} +\pder[T]{z}\der[z]{t} = \nabla T\cdot\vr'(t),\tag{1} \end{align*} where $\nabla T = \langle \p T/\p x,\p T/\p y,\p T/\p z\rangle$ is the gradient of temperature. Assume that $|\vr'(t)|=1$, independent of path (that is, the paths are parametrized by arclength). The derivative $dT/dt$ is then a measure that can be used to compare the rate of change of $T$ for different paths through $\vr_0$. Note that $$\left|\pder[T(\vr(t))]{t}\right| = |\nabla T|\cos\th,$$ where $\th$ is the angle between $\nabla T$ and $\vr'(t)$. The magnitude of the rate of change is clearly maximized for $\th=0,\pi$. The value of $T$ does not change for paths with $\th=\pi/2$. These paths are tangential to the equipotential at $\vr_0$. Thus, $dT/dt$ is maximally negative if $\vr'$ is antiparallel to $\nabla T$ (and thus perpendicular to the equipotential). Since $\vr'$ is tangential to the path, we expect $T$ to decrease maximally in the direction given by $-\nabla T$. Thus, we let $\Delta\vr = -\varepsilon\nabla T$, where $\varepsilon>0$ is sufficiently small.

Example

If $T(\vr) = x^2+y^2+z^2$, the equipotentials are spheres and $\nabla T = 2\langle x,y,z\rangle = 2\vr$. Thus, $\Delta r = -2\varepsilon \vr_0$. This displacement is directly towards the origin, where the temperature has a minimum value of $T({\bf 0}) = 0$. See the figure below.

Figure 1. Equipotentials and displacement vector.

One dimension

Equation (1) and its interpretation generalize nicely to higher and lower dimensions. In one dimension we have $$\der[T(x(t))]{t} = \der[T]{x} \der[x]{t}.$$ Clearly this value is maximally negative if $dT/dx$ and $dx/dt$ are opposite in sign. Thus, we let $\Delta x = -\varepsilon dT/dx$.

Example

If $T(x) = x^2$, we find $\Delta x = -2\varepsilon x_0$. This displacement is again directly towards the minimum at $x=0$, independent of the sign of $x_0$.

Answer 5

Multidimentional optimization task can be presented as the sequence of $\;\text{1D}\;$ tasks.

Really, let we start from the point $\;z=A,\;$ wherein $\text{ grad } f(z)\bigg|_{z=A} = \overrightarrow {g_A}.\;$ Assuming $\;\overrightarrow{\Delta z} = \overrightarrow{g_A}t,\; $ we get $\;\text{1D}\;$ optimization task for the variable $\;i.$, The final point $B=A+\overrightarrow{g_A}\,t_{opt}\;$ of the current optimization task assumed as the starting point for the next task, with the suitable $\;\text{1D}\;$ vizualization of the each step.

Answer 6

When I think about a function $f: \mathbb R^n \to \mathbb R$, I can only visualize the cases where $n = 2$ or $n = 3$ (or $n = 1$, of course). I like to imagine that $f$ tells us the temperature at each point in space. A mosquito who likes cool temperatures is flying around. Which way should it move? The gradient vector at its current location points in the direction in which the temperature is increasing most rapidly. So the mosquito should move a little bit in the opposite direction, which is the direction in which the temperature is decreasing most rapidly. Then the mosquito should recompute the gradient of $f$ at its new location, and move a bit again in the negative gradient direction. And so on. This is gradient descent.

Answer 7

Well, I don't believe that it is easy to maintain an accurate model of a $>3$ dimensional space in one's imagination for sustained periods, but one can certainly gain an intuition for each $3$D cross-section and maneuver around that way. Pick easy shapes, like spheres or tetrahedrons, to try and visualize. The issue is mostly visualization, since our minds are in constant interaction with multidimensional spaces in day-to-day life, though those spaces are simply processed less visually.

Dreams, for example, compose many dimensions of the waking life and compress them into highly dynamic visualizations of usually $<3$ dimensions. As for generalization, typically I take a linear variable simply map it to multilinear vectors or parametric spaces. It's helpful to know that the intersection of $k$ $n$ dimensional surfaces is a $n-k+1$ dimensional surface in $n$ space, for example.

It is fundamentally very difficult to communicate understanding, so the best advice is to simply persist at trying to visualize these spaces in your own unique way. There are some who have entertained multidimensional spaces in their work for so long that they are simply, precisely familiar with $4$D (and up) landscapes. The more you persist, the more dimensions you will add. Sometimes, after a lifetime of persistence, you only end up with a half a dimension more than three. Other times, you go all the way to eight. It depends.

So, the best method, I think, is to pick simple shapes that are easy to generalize into higher dimensions (like spheres) and visualize each part until you can put it together. Then slowly expand your horizons. Manipulate the sphere here or there to change it this way or that. Soon enough what seems impossible to many will be an average consideration for you.