Firstly, the original numbers must be "larger" than $\{5,6,7,8\}$ since we will subtract 6 and 7.
Depending on what kind of class this quiz took place, there might be shortcuts where one can make some more "educated guesses".
Observe that the subtraction of $\{2,6,7,2\}$ are all small, so they cannot create large differences (and we have positive numbers to begin with). A reasonable guess is that the resultant geometric progression cannot possibly have a large common ratio $r$. One might try out some nice small whole numbers like $r = 2$.
You write down guesses of geometric progression for different initial terms like $\{1, 2, 4, 8\}$, $\{2, 4, 8, 16\}$, and $\{3, 6, 12, 24\}$, then reverse the operation and add $\{2,6,7,2\}$ to see if they are arithmetic.
Voila, you see that $\{3, 6, 12, 24\}$ works nicely, coming from an original arithmetic progression $\{5, 12, 19,26\}$.
In general, the following is how one solves the question regardless if the numbers are nice.
Denote the original 4 numbers in arithmetic progression as $\{a, a+d, a+2d, a+3d\}$, then the geometric progression (with same initial $a$ and ratio $r$) after the subtraction gives
\begin{align*}
r = \frac{ a + d - 6 }{ a -2 }\quad\text{as Eq.(1),} &&
r = \frac{ a + 2d - 7 }{ a + d - 6 } \quad\text{as Eq.(2),} &&
r = \frac{ a + 3d - 2 }{ a + 2d - 7} \quad\text{as Eq.(3)}
\end{align*}
It is the common $r$ for all 3 equations by definition. We have 3 unknowns $\{a, d, r\}$ and 3 equations. Let's solve them.
Cross multiply Eq.(1) and Eq.(2)
\begin{align*}
(a-2)(a+2d-7) &= (a + d -6)^2 \\
a^2 + 2ad -9a - 4d + 14 &= a^2 + d^2 +36 -12 a - 12 d + 2 a d \\
\color{magenta}{3a} &= \color{magenta}{d^2 - 8d + 22}
\end{align*}
Cross multiply Eq.(2) and Eq.(3)
\begin{align*}
(a+d-6)(a+3d-2) &= (a + 2d -7)^2 \\
a^2 + 3d^2 + 12 + 4ad -8a - 20d &= a^2 + 4d^2 + 49 -14 a - 28 d + 2 a d \\
\color{magenta}{6a} &= \color{magenta}{d^2 - 8d + 37}
\end{align*}
The two magenta equations give $3a = 15$ therefore $a = 5$.
Substituting back to either one of the magenta equations we get $d^2 - 8d +7 = 0 \implies (d-1)(d-7)=0$, hence $d = 1$ or $d = 7$
If $d = 1$, then the original arithmetic sequence is $5,6,7,8$, which after the subtraction is $\{3,0,0,6\}$. This is clearly not a solution. The $d = 1$ is extraneous where $a+d-6=0$ and $a+2d-7 = 0$ invalidate the cross multiplication.
For $d = 7$, we have the original arithmetic progression as $\{5, 12, 19,26\}$, which upon the subtraction is the geometric progression $\{3, 6, 12, 24\}$. This is our solution.