I need to find gcd of $x^4-3x^3-2x+6$ and $x^3-5x^2+6x+7$ in $\mathbb Z/7 \mathbb Z[x]$, the integer polynomials mod $7$. Please any help will be appreciated.
In general, you use the same method as gcd of numbers: the Euclidean algorithm (detailed below). In this specific (i.e. non-general) case you can take a short-cut: since you see that the constant term in your second polynomial is $0$ mod $7$, we know that $0$ is a root mod $7$. So the second polynomial factors as $(ax^2+bx+c)x$, and we might hope there's another root. Testing, we see that $x=2,3$ are also roots, so the factorization is
Checking the first polynomial for common roots shows only $x=3$ is a common root, so $(x+4)$ is the gcd.
The more general way:
$$x^4-3x^3-2x+6=x(x^3-5x^2+6x+7)+(2x^3+x^2-2x-1)\mod 7$$ $$x^3-5x^2+6x+7=4(2x^3+x^2-2x-1)+(-2x^2+4)\mod 7$$ $$2x^3+x^2-2x-1=(-x-4)(-2x^2+4)+(2x+1)\mod 7$$ $$-2x^2+4=(-x)(2x+1)+(x+4)\mod 7$$ $$2x+1=2(x+4)+0\mod 7$$
so the gcd is $x+4$ since $2$ is a unit, though you can write $2(x+4)$ if you want, as well.