Why is a straight line the shortest distance between two points? The first application I was shown of the calculus of variations was proving that the shortest distance between two points is a straight line. Define a functional measuring the length of a curve between two points:
$$
I(y) = \int_{x_1}^{x_2} \sqrt{1 + (y')^2}\, dx,
$$
apply the Euler-Langrange equation, and Bob's your uncle.
So far so good, but then I started thinking: That functional was derived by splitting the curve into (infinitesimal) - wait for it - straight lines, and summing them up their lengths, and each length was defined as being the Euclidean distance between its endpoints*. 
As such, it seems to me that the proof, while correct, is rather meaningless. It's an obvious consequence of the facts that (a) the Euclidean norm satisfies the triangle inequality and (b) the length of a curve was defined as a sum of Euclidean norms.
Getting slightly philosophical, I would conjecture that proving that the shortest distance between two points is a straight line is looking at things the wrong way round. Perhaps a better way would be to say that Euclidean geometry was designed to conform to our sensory experience of the physical world: the length of string joining two points is minimized by stretching the string, and at that point, it happens to look/feel straight.
I'm just wondering whether people would agree with this, and hoping that I may get some additional or deeper insights. Perhaps an interesting question to ask to try to go deeper would be: why does a stretched string look and feel straight?

*: To illustrate my point further, imagine we had chosen to define the length of a line as the Manhattan distance between its endpoints. We could integrate again, and this time it would turn out that the length of any curve between two points is the Manhattan distance between those points.
 A: Let me start by saying that on a gut level, I agree with everything you said. But I feel like I should make this argument anyway, since it might help you (and me!) sort out ideas on the matter.

It doesn't seem inconsistent to argue that the model of Euclidean space (defined by, say, the Hilbert axioms) as $\Bbb R^n$ really gets around all the philosophical questions. We can ask why $\Bbb R$ and such, but taken as an object in its own right, the standard inner product defines everything from the geometry to the topology to the notion of size.
In this view, the integral you mentioned can be taken as the definition of "length" of a curve (in $\Bbb R^2$, I think), observing that it matches with the Lebesgue measure when the curve under consideration is given by an affine transformation (although this is formally irrelevant). The definition is motivated not as being broken down into straight lines, but rather into vectors, which have a different definition of length (this does not trouble me much: it is only wishful thinking that we use the same term for each). The notion of a "line" per se arises as a fairly natural question: what is the infimum of the length between two points and if so, is there actually a curve that achieves it? Once you see that not only is the answer "yes" but also "and it's unique", it's not much of a stretch to think these objects are worth adding to our basic understanding of the space.
As for the remark choosing the Manhattan distance: nothing prevents you from doing this, but if you prefer this to be your norm (which you well might, for the reasons you described above), then you lose all aspects of geometry relating to angles. You also lose the uniqueness of minimal-length curves, and perhaps you then become less interested in the question. From the omniscient perspective, we might see this as a tragedy, an acceptable loss, or even as a gain. This objection, as well as Will Jagy's comment, only appear to highlight the flexibility we have in terms of which formalisms to use.
Your other question is of course much harder to answer, but I think a nice reduction of the question is "What makes $\Bbb R^3$ the most physical-feeling model?" The question is particularly interesting in light of the fact that $\Bbb R^3$ is certainly not a complete model of space for actual physics! But I do not think you would be taken seriously if you tried to argue that the universe is not a manifold. For some reason, (open subsets of) $\Bbb R^n$ is locally "almost right".

This is just me talking out of you-know-where: It could be that the reason we have such strong intuitions about straightness and distance is because of evolutionary pressures. People who could intuit how to get from place to place efficiently would not burn the unnecessary calories, and in a less sheltered world this could help them reach the age of sexual viability. Once we began thinking inductively then we would be allowed to think of the notion of straightness as going on forever, and as an general construct rather than a situational feature. But by then it would be too late to straighten out the conflation of straightness and linearity, and we would need to wait a long time before we could do so with any rigor.
A: I think a more fundamental way to approach the problem is by discussing geodesic curves on the surface you call home. Remember that the geodesic equation, while equivalent to the Euler-Lagrange equation, can be derived simply by considering differentials, not extremes of integrals. The geodesic equation emerges exactly by finding the acceleration, and hence force by Newton's laws, in generalized coordinates.
See the Schaum's guide Lagrangian Dynamics by Dare A. Wells Ch. 3, or Vector and Tensor Analysis by Borisenko and Tarapov problem 10 on P. 181
So, by setting the force equal to zero, one finds that the path is the solution to the geodesic equation. So, if we define a straight line to be the one that a particle takes when no forces are on it, or better yet that an object with no forces on it takes the quickest, and hence shortest route between two points, then walla, the shortest distance between two points is the geodesic; in Euclidean space, a straight line as we know it.
In fact, on P. 51 Borisenko and Tarapov show that if the force is everywhere tangent to the curve of travel, then the particle will travel in a straight line as well. Again, even if there is a force on it, as long as the force does not have a component perpendicular to the path, a particle will travel in a straight line between two points.
Also, as far as intuition goes, this is also the path of least work.
So, if you agree with the definition of a derivative in a given metric, then you can find the geodesic curves between points. If you define derivatives differently, and hence coordinate transformations differently, then it's a whole other story.
A: There are several issues here:


*

*A typo in your equation. Your $y'$ should be squared. That is $y'^2$.
(corrected).

*Nothing stops you or me to use other norms. The norm 1 could have issues, since we need the functions $y$, and the "Lagrangian" to be twice continuously differentiable before deriving the Euler-Lagrange equations.

*Let us assume you use an $L_4$ norm. Then define the length of a
    path as $\lim_{ | \Delta t | \to 0} \sum  \| \Delta \bf{x}_i \|_4$
    where by $\lim | \Delta t |$ I mean the size of a partition from in
    a real interval $[a,b]$ where the parametric function $\bf{x}(\it
        t)=(x(t),y(t))$ is defined. Then using the same idea of
    differentials you can say $ds = (dx^4 + dy^4)^{1/4}$ and end up with
    another equation $I(y)=\int_A^B (1 + y'^4)^{1/4} dx$. This is
    perfectly fine. You could minimize this cost function, but if you
    apply the Euler-Lagrange equation here you would get:
     \begin{equation}      \frac{d}{dx} \frac{y'^3}{(1+y'^4)^{1/4}}=0 \end{equation} 
       and then $y'^3/(1+y'^4)^{1/4}=\text{constant}\equiv C$. The
        solution in    this case is not a straight line.   So what is the
        point? 
         It is an optimization problem and the solution and interpretation changes according to the type of norm and space that
you consider. If a distance in kilometers is computed in meters we
need to  multiply by 1000 to compensate for the fact that the new metric
is now
        1/1000 of the old metric. A straight segment in a plane would be deformed (bent or stretched)
        when changing from Euclidean metric to metric such as $L_4$, to adjust to the new shortest path condition. There are two different things that change the solution of the
    problem. The curvature of the space (manifold) being used, and the
    metric that you want to impose. From differential geometry the
    metric tensor $g_{ij}$ is the natural (Riemannian) metric to impose to a curved
    space and this metric generates the geodesics of the system. That
    is, the paths of least distances on that metric. Again, you are free
    to choose another metric, but then those paths will be deformed
    accordingly.

*Now, here is a question that I do not see many people (books, documents) asking. It is obvious that we are finding a minimum but this should be proven. That is, is the cost (objective) function convex? 
In other words, the solution that the shortest path between two points is straight in $L_2$ norm here in a plane is a necessary but not sufficient condition for stationarity. We still need to prove the "sufficient" part and then ask if the minimum is a local or global  minimum, if it is unique. Seems obvious but how to prove this?

*Here is an interesting argument which does not require smooth constraints and the sofisticated Euler-Lagrange machinery. 
We search for the shortest path length $L(A,B)$ between two points $A$ and $B$. The distance is an invariant under translation and rotations (those are isometries). Then without loss in generality we can think that $A=(0,0)$ the origin and $B=(0,\ell)$, where $\ell$ is the straight distance between $A$ and $B$. Then the length of any segment between $A$ and $B$ is given by
\begin{equation}
L(A,B) = \int_0^{\ell} \sqrt{1+ y'^2} dx \ge \int_0^{\ell} dx = \ell.
\end{equation}
since $y'^2 \ge 0 $. So $\ell$ is a lower bound for lengths over all
qualifying segments. Even more, the straight distance is $\ell$, so
$\ell$ is the infimum, the minimum, and the shortest distance.
If you pick a metric inherited from $L_4$ norm, you would get also,
doing the same analysis that $\ell$ is a lower bound of all qualifying
paths between $A$ and $B$, but this time $\ell$ is smaller than the
possible shortest measure under this new metric.
So, metrics induced by the norm
$L_n$, $n > 2$, create longer shortest paths (at least for $n$ even,
what happens if $n$ is odd?). This makes sense in the
case of the sphere, where a straight segment is shorter than a geodesic on the sphere. 


It is obvious that there is not upper bound for the distance. You give me a distance and I can always find a path which is longer. The question is: does the distance has a lower bound? The answer is "yes" since distance is always greater or equal to zero, so it has an "inf".  Still, if the infimum is unique this does not mean that the path is unique. Think of a distance on a unit sphere between two antipodes. There are an infinite number of paths which satisfy the shortest path of length $\pi$. In the case of a path on a plane we can use Euclid's postulate: There is only one line through two points. So it is unique. It rest to prove: Is the infimum the length of the line between those two points? 
For an excellent discussion on the topic of this question please check:
A course in Metric Geometry  chapter 2.
A: my attempt by using Euler-lagrange equations :
$$let \ the \ line \ between\  two\ points\  S\\
\\
\\
\therefore s=\int_{x_{1}}^{x_{2}}ds\ \ \ \ \ ,\ \ \because \ (ds)^2=(dx)^2+(dy)^2\\
\\
\\
\therefore ds=\sqrt{(dx)^2+(dy)^2}=\sqrt{1+(\frac{dy}{dx})^2}\ dx\\
\\
\\
\therefore S=\int_{x_{1}}^{x_{2}}\sqrt{1+y'^2}\ dx\ \ \ \ \ ,\ \ let\ F=\sqrt{1+y'^2}\\
\\
\\$$
since the condition to be S min limit is :
$$\frac{d}{dx}(\frac{\partial F}{\partial y'})=\frac{\partial F}{\partial y}$$
$$**EULER -LAGRANGE\  EQUATION**$$
now we find 
$$\\because \ \frac{\partial F}{\partial y}=0 \ \ \ \ , \frac{\partial F}{\partial y'}=\frac{y'}{\sqrt{1+y'^2}}\\$$
\
$$\therefore \frac{d}{dx}\left ( \frac{\partial F}{\partial y'} \right )-\frac{\partial F}{\partial Y}=0\ \ \ \ \ \Rightarrow \frac{d}{dx}\left ( \frac{y'}{\sqrt{1+y'^2}} \right )=0\\$$
\
\
$$\therefore \frac{y'}{\sqrt{1+y'^2}}=c\ \ \ \ \ \ \Rightarrow y'^2=c^2(1+y'^2)\\
\\
\\
\therefore y'^2(1-c^2)=c^2\ \ \ \ \Rightarrow y'^2=\frac{c^2}{1-c^2}\ \ \ \ \ \ \ , \ \ let\ \ a^2=\frac{c^2}{1-c^2}\\$$
\
$$\therefore y'^2=c^2\ \ \ \ \ \ \Rightarrow y'=\pm c\ \ \ \ \ \ \Rightarrow \frac{dy}{dx}=\pm c\\$$
\$$
\therefore y=\pm x\ +b\$$     
and that is the equation of the straight line
