What is the adjoint representation of $\operatorname{GL}_n(\mathbb{R})$? This question has been asked several times, and virtually every introductory textbook on Lie groups and Lie algebras will cover this. But every single explanation goes beyond me.
Let $G$ be a Lie group. Let $\Psi : G \to \operatorname{Aut}(G)$ be the map sending $g \in G$ to the conjugation map $h \mapsto g h g^{-1}$. For any $g \in G$, the conjugation map $\Psi(g) : G \to G$ admits a total derivative $d\Psi(g)_e : T_e G \to T_e G$. Varying $g$, this may be expressed as a representation $\operatorname{Ad} : G \to \operatorname{Aut}(T_e G)$. Then, we define $\operatorname{ad} : T_e G \to \operatorname{End}(T_e G)$ to be the total derivative $d \operatorname{Ad}_e$. 
Question. What is $\operatorname{ad}$ when $G = \operatorname{GL}_n(\mathbb{R})$?
What I know. Suppose $A$ is a matrix in $\operatorname{GL}_n(\mathbb{R})$. Then $\Psi(A)$ is the map sending a matrix $M$ to $AMA^{-1}$. We can take the derivative of $\Psi(A)$, which is a map $d \Psi(A)_e : T_e \operatorname{GL}_n(\mathbb{R}) \to T_e \operatorname{GL}_n(\mathbb{R})$. As $\operatorname{GL}_n(\mathbb{R})$ is an open submanifold of $\operatorname{Mat}_n(\mathbb{R})$, I can identify the tangent space of $\operatorname{GL}_n(\mathbb{R})$ with $\operatorname{Mat}_n(\mathbb{R})$. Moreover, for every matrix $X$, I have a particularly simple choice of a path $\gamma : I \to \operatorname{GL}_n(\mathbb{R})$ passing through $I_n$ with derivative $X$ at $t = 0$, namely, $\gamma(t) = I_n + tX$.
So to find out what $\operatorname{Ad}(A)$ is, I can write $d\Psi(A)_e(X)$, for any matrix $A$ in $\operatorname{GL}_n(\mathbb{R})$ and for any element $X \in T_e \operatorname{GL}_n(\mathbb{R}) = \operatorname{Mat}_n(\mathbb{R})$, as
$$d \Psi(A)_e(X) = \frac{d}{dt} \Psi(A)(I + tX) = \frac{d}{dt} A(I + tX)A^{-1} = \frac{d}{dt} (I + AtX A^{-1}) = AXA^{-1}.$$
So, $\operatorname{Ad}(A)$ is $d\Psi(A)_e$, which is the map $X \mapsto AXA^{-1}$.
But then you have to take a further derivative. And here the limit of my capacity is reached. The complexity of the objects and maps involved simply impede me from seeing what it even means to take a derivative anymore, let alone being able to calculate it.
 A: I would have guessed this was a duplicate, but the closest match I could find was: Definition of differential of Adjoint representation of Lie Group.
In any case you're almost there:
Hint As is common in this context, denote $\mathfrak{gl}(n, \Bbb R) := T_I \operatorname{GL}(n, \Bbb R) \cong \operatorname{Mat}_n(\Bbb R)$; since $\operatorname{GL}(n, \Bbb R)$ is a matrix group, I've used the symbol $I$ for its identity element instead of the generic symbol $e$.
As you write, under the identifications you made, the adjoint representation of $\operatorname{GL}(n, \Bbb R)$ is
$$\operatorname{Ad} : \operatorname{GL}(n, \Bbb R) \to \operatorname{Aut}(\mathfrak{gl}(n, \Bbb R)), \qquad \operatorname{Ad}(A)(X) = AXA^{-1} .$$
Then, by definition, to compute the adjoint representation
$$\operatorname{ad}: \mathfrak{gl}(n, \Bbb R) \to \operatorname{End}(\mathfrak{gl}(n, \Bbb R)) $$ of $\mathfrak{gl}(n, \Bbb R)$ we just differentiate $\operatorname{Ad}(A)(X)$ with respect to $A$.
So, fix a tangent vector $B \in \mathfrak{gl}(n, \Bbb R) \cong \operatorname{Mat}_n(\Bbb R)$, substitute the expression $I + t B$ for $A$ (which defines a path $t \mapsto I + t B$ in $\operatorname{GL}(n, \Bbb R)$ with tangent vector $B$ at $t = 0$) in $\operatorname{Ad}(A)(X) = AXA^{-1}$, and differentiate with respect to $t$ at $t = 0$.

 \begin{align}\operatorname{ad}(I + t B)(X) &= \left.\frac{d}{dt}\right\vert_{t = 0} \operatorname{Ad}(I + t B)(X) \\ &= \left.\frac{d}{dt}\right\vert_{t = 0} [(I + t B) X (I + t B)^{-1}] \\ &= (B) X (I) + (I) X \left.\frac{d}{dt}\right\vert_{t = 0}[(I + t B)^{-1}] .\end{align} Differentiating $(I + t B) (I + t B)^{-1} = I$ and evaluating at $t = 0$ gives that $\left.\frac{d}{dt}\right\vert_{t = 0} [(I + t B)^{-1}] = -B$, and so substituting gives $$\color{#df0000}{\boxed{\operatorname{ad}(B)(X) = B X - X B}} .$$ Under the identifications you've made, this expression is the commutator $[B, X]$ in the ring $\operatorname{Mat}_n(\Bbb R)$.

A: Yet another answer - but as I was typing it up:
As you say, one can identify the tangent space $T_e(G)$ with $T={\rm Mat}_n(\mathbb R)$. Let  $Y$ an element of $T$, and one can consider $\gamma(t) = \exp t Y$, which is a path through $e$ in ${\rm GL}_n(\mathbb R)$ (or one can use your choice of $\gamma$ - I chose the exponential because of tradition and because it's defined for all $t$ - but who cares). As you say, $G$ acts on $T$. Thus, 
$$ t \mapsto g(t)={\rm Ad}(\gamma(t))$$ is a curve in ${\rm Aut }(T)$. So if $X$ is a fixed element in $T$, then 
$$ t \mapsto g(t)(X)= {\rm Ad}(\gamma(t))(X)=\gamma(t) X \gamma(-t)$$
is a curve in $T$, which we can differentiate at $t=0$: 
$$ g'(0)(X)={d\over dt}\Big\vert_{t=0}{\rm Ad}(\gamma(t)) (X) ={d\over dt}\Big\vert_{t=0} \gamma(t) X \gamma(-t).$$
By the product rule, one gets
$$ g'(0)(X) = {\rm ad} (Y) (X)= YX -XY.$$
A: Question. What is $\operatorname{ad}$ when $G = \operatorname{GL}_n(\mathbb{R})$?
It is the adjoint representation of the Lie algebra of $G$, which is the general linear Lie algebra $\mathfrak{gl}_n(\Bbb R)$, with Lie bracket
$[A,B]=AB-BA$. 
Here $\operatorname{ad}(A)$ is defined by 
$$
\operatorname{ad}(A)(B)=[A,B]=AB-BA.
$$
There are several posts on this site, which deal with this question in some way. Here are a few references:
Lie algebras of GL(n,R) and differentials
The Lie bracket of $\mathfrak{gl}_n(\mathbb{R})$ is the matrix commutator
$\mathsf{GL}(n, \mathbf{R})$ Mapping and Derivative
