When is $\operatorname{Spec} A^G \cong (\operatorname{Spec} A)/G$ true? Let $G$ be a group acting on a ring $A$. I would like to know in which generality we know that $\operatorname{Spec} A^G \cong (\operatorname{Spec} A)/G$. Moreover, when this is true, it also holds for the underlying topological spaces?
I know that both facts hold when $G$ is finite (but I don't have a proof and I would be grateful if someone sketched one or commented where I can find it). But I'm also interested in more general groups. In particular, I wonder if it holds for profinite groups.
Edit: let me be more precise.
If $G$ is a finite group we can define a scheme $(\operatorname{Spec} A)/G$ whose underlying topological space is the usual quotient and whose structure sheaf is simply $(\pi_*\mathcal{O}_{\operatorname{Spec}A})^G$, where $\pi$ is the canonical projection. This satisfies a universal property in the category of schemes. In this case, I don't know how to prove that this scheme is isomorphic to $\operatorname{Spec}A^G$.
For infinite groups, the quocient as defined above need not be a scheme. But it exists nevertheless as a topological space. In this case, it is still true that the topological space $(\operatorname{Spec} A)/G$ is homeomorphic to $\operatorname{Spec}A^G$?
 A: To answer your question it is useful, as some users have already pointed out, to make precise what we mean by ‘quotient’. The book [MFK] is very useful for this, and so I will recall the relevant parts here.
The first is a definition of quotient which is the most naive from the perspective of functor of points and, roughly, says that a map $\pi\colon Y\to X$ is a ‘quotient by $G$‘ (where $G$ is some group acting on $Y$) if $X$ satisfies the correct moduli problem.

Definition 1 ([MFK, Definition 0.5]): Let $S$ be a base scheme, $Y$ an $S$-scheme, and $G$ an $S$-group scheme acting on $Y$. Then, a
morphism $\pi\colon Y\to X$ of $S$-schemes is called a categorical
quotient if

*

*the diagram $$\begin{matrix}G\times_S Y & \xrightarrow{\text{act.}} & Y\\ \downarrow & & \downarrow\\ Y &
 \xrightarrow{\pi} & X\end{matrix}$$ commutes (where the left vertical
arrow is the natural projection, and the right vertical arrow is the
map $\pi$).

*For any $S$-scheme $Z$ the natural map  $$\mathrm{Hom}_S(X,Z)\to\mathrm{Hom}_S(Y,Z)$$ is injective with image
$\mathrm{Hom}_S(Y,Z)^G$ (where this is shorthand for the set of maps $\varphi\colon Y\to Z$ of $S$-schemes $Y\to Z$ such the diagram in 1. is commutative with $\pi\colon Y\to X$ replaced by $\varphi\colon Y\to Z$).


In words, this definition says that $\pi\colon Y\to X$ is a ‘categorical quotient’ if it’s 1) $G$-invariant (where $X$ is given the trivial $G$-action), and 2) a map $\varphi\colon Y\to Z$ factorizes through $\pi\colon Y\to X$ if and only if $\varphi$ is ‘$G$-equivariant’ (where $Z$ is given the trivial $G$-action). In short, a categorical quotient $\pi\colon Y\to X$ is an initial $G$-equivariant morphism to an $S$-scheme with trivial $G$-action.
This definition is very conceptually simple, but is really not super useful from a concrete perspective—-if categorical quotient is to exist, the definition is not very illuminating as to what it must ‘look like’. To this end, Mumford also defines an a priori more more topological, concrete guess of what a quotient must be.

Definition 2 ([MFK, Definition 0.6]): Let $S$ be a scheme, $Y$ an $S$-scheme of finite type, and $G$ an $S$-group scheme of finite type
acting on $Y$. A morphism of schemes $\pi\colon Y\to X$ is a
geometric quotient if

*

*$\pi$ is $G$-equivariant (as in 1. of Definition 1),

*the map $\pi$ is surjective and for all geometric points $\mathrm{Spec}(K)\to S$ (i.e. maps to $S$ from the spectra of
algebraically closed fields) the fibers of the map $Y(K)\to X(K)$ are
precisely the $G(K)$-orbits of $G(K)$ acting on $Y(K)$,

*the map of underlying topological spaces of $\pi$ is a quotient map (i.e. for $U\subseteq X$ one has that $\pi^{-1}(U)$ is open if and
only if $U$ is open),

*the natural map $\mathcal{O}_X\to \pi_\ast\mathcal{O}_Y$ induced by $\pi$ is an isomorphism onto the subsheaf $(\pi_\ast
\mathcal{O}_Y)^G$ which, by definition, has sections over an open
$U\subseteq X$ given by the set of elements
$f\in\mathcal{O}_X(\pi^{-1}(U))=\mathrm{Hom}_S(\pi^{-1}(U),\mathbf{A}^1_S)$
which are $G$-equivariant as in 1. of Definition 1.


This definition is the ultimate concrete realization of what a quotient should be. Correctly interpreted a geometric quotient $\pi\colon Y\to X$ (if it exists) is equal to $(|Y|/G,(\pi_\ast \mathcal{O}_Y)^G)$—the topological quotient of $Y$ by the $G$-action with the sheaf of ‘invariant sections’.
What do I mean by ‘correctly intepreted’? Well, one must be minorly careful since, of course, the action of an $S$-group scheme $G$ on a $Y$ doesn’t mean that there is a group acting on the topological space $Y$ in any naive sense—right? But, recall that there is a natural identification of sets
$$Y=\left\{\text{Geometric points }\mathrm{Spec}(K)\to Y\right\}/\sim\qquad (1)$$
where two geometric points $\mathrm{Spec}(K_i)\to Y$ for $i=1,2$ are equivalent (what I’m writing as $\sim$) if there exists some third algebraically closed field $L$ containing $K_1,K_2$ such that compositions $\mathrm{Spec}(L)\to\mathrm{Spec}(K_i)\to Y$ are equal for $i=1,2$. With Equation $(1)$ we may define an equivalence relation $\simeq$ on $Y$ by saying that two geometric points $\mathrm{Spec}(L)\to G$ are equivalent if they are in the same $G(L)$-orbit. Then, when I write $|Y|/G$ what I mean is the quotient topological space $Y/\simeq$. Note that in the special case when $G=\underline{G}$ is a constant finite group scheme (I am abusing the abstract topological group and the group scheme) then $G$ does act on $Y$ and $|Y|/G$ (as I’ve defined above) does agree with the naive definition of $|Y|/G$.
So then, a natural set of questions is the following:

*

*in what generality do categorical quotients exist,

*in what generality are these categorical quotients actually geometric quotients (NB: a geometric quotient is automatically a categorical quotient by [MFK, Proposition 0.1])?

Of course, as your question indicates, you are interested in the case when $Y$ is actually an affine scheme $\mathrm{Spec}(A)$.
In this case, this question is answered in brilliant form in [MFK] in the case when $S=\mathrm{Spec}(k)$ (where $k$ is a field, not necessarily algebraically closed).

Theorem 1 ([MFK,Theorem 1.1 (and its proof), and Amplication 1.3]) Let $G$ be a reductive group over $k$ $(\ast\ast)$ and $Y=\mathrm{Spec}(A)$ an
affine scheme over $S=\mathrm{Spec}(k)$ of finite type. Then,

*

*the natural map $\mathrm{Spec}(A)\to\mathrm{Spec}(A^G)$ is a categorical quotient (here $A^G$ is the subring of $A$ consisting of
those elements $f$ which, when interpreted as maps
$Y\to\mathbf{A}^1_k$, are $G$-equivariant as in 1. of Definition 1),

*this map is a geometric quotient if and only if for any geometric point $\overline{x}\colon \mathrm{Spec}(K)\to Y$ the orbit $G(K).
 \overline{x}\subseteq Y_{K}$ is closed.


This theorem is absolutely beautiful. As a special case we obtain an answer to your first question.

Corollary 1: Let $G$ be a finite abstract group (of order coprime to the characteristic of $k$) acting on the affine $k$-scheme
$\mathrm{Spec}(A)$. Then, $\mathrm{Spec}(A)\to\mathrm{Spec}(A^G)$ is a
geometric quotient (thus, a fortiori, $\mathrm{Spec}(A^G)$ is
$\mathrm{Spec}(A)/G$ with the sheaf of invariant sections).

This is clear since the closedness condition in 2. of Theorem 1 is automatic since this orbit will be a finite set of $K$-points of $Y_K$ and so automatically closed.
But, you can also use this to give other wonderfully interesting examples.
Example 1: Let $\mathbf{G}_{m,k}$ act on $\mathrm{GL}_{n,k}$ in the obvious way (by identifying $\mathbf{G}_{m,k}=Z(\mathrm{GL}_{n,k})$). Then, by the above the geometric quotient of $\mathrm{GL}_{n,k}$ by $\mathbf{G}_{m,k}$ exists and is equal to $\mathrm{Spec}(\mathcal{O}(\mathrm{GL}_{n,k})^{\mathbf{G}_{m,k}})$. I leave it for you to calculate that this is precisely $\mathrm{PGL}_{n,k}$ as you’d expect!
As for your question about infinite constant groups, the answer is definitely no in full generality that $\mathrm{Spec}(A^G)$ has underlying topological space $\mathrm{Spec}(A)/G$.
Example 2: Consider $\mathbf{Z}$ acting on $\mathbf{C}[x]$ by $n\cdot x=x+n$ then $\mathbf{C}[x]^\mathbf{Z}$ is the set of polynomials $f(x)$ such that $f(x)=f(x+n)$ for all $n$, and evidently this is the ring $\mathbf{C}$. But, $\mathbf{A}^1_\mathbf{C}/\mathbf{Z}$ is not a point.
For profinite groups it’s more possible (assuming that finite quotients have order invertible in $k$) by trying to bootstrap from the finite case and commuting some sort of limits—haven’t thought too much. In the positive direction though, we have the following examlpe.
Example 3: Let $K$ be a field and set $\Gamma:=\mathrm{Gal}(K^\mathrm{sep}/K)$. Let $A$ be a finite type $K$-algebra. It is true that the natural map
$$\mathrm{Spec}(A_{K^\mathrm{sep}})\to \mathrm{Spec}(A)=\mathrm{Spec}(A_{K^\mathrm{sep}}^\Gamma)$$
identifies the target, topologically, with $\mathrm{Spec}(A_{K^\mathrm{sep}})/\Gamma$.

EDIT: Here’s a proof of Example 3–it’s WAY overkill, sorry I’m just writing a quick justification. Note that by Tag 0383 the map
$\mathrm{Spec}(K^\mathrm{sep})\to \mathrm{Spec}(K)$ is universally
open and surjective. Thus,
$\mathrm{Spec}(A_{K^\mathrm{sep}})\to\mathrm{Spec}(A)$ is open and
surjective, and thus generalizing (see Tag 040F) and surjective.
Thus, by standard theory of spectral spaces, the map
$\mathrm{Spec}(A_{K^\mathrm{sep}})\to\mathrm{Spec}(A)$ is a quotient
map (see [Scholze, Lemma 2.5]). Note that this map is evidently
equivariant for the action of $\Gamma$ and thus we get an induced map
$\mathrm{Spec}(A_{K^\mathrm{sep}})/\Gamma\to \mathrm{Spec}(A)$. This
map is clearly still a quotient map. But, it is also a bijection (cf..
[GW, Proposition 5.4]). Thus, it’s a homeomorphism.
EDIT EDIT: A more elementary way to verify what I said is the following. Note that $\mathrm{Spec}(A_{K^\mathrm{sep}})=\varprojlim
> \mathrm{Spec}(A_L)$ where $L$ runs over the finite Galois
subextensions of $K^\mathrm{sep}/K$. The action of $\Gamma$ on
$\mathrm{Spec}(A_{K^\mathrm{sep}})$ may be seen as the induced action
of $\varprojlim \mathrm{Gal}(L/K)$ on $\varprojlim \mathrm{Spec}(A_L)$
and so, consequently, I leave it to you to verify that the map
$\mathrm{Spec}(A_{K^\mathrm{sep}})\to\mathrm{Spec}(A)$ may be
identified as the projective limit of the maps
$\mathrm{Spec}(A_L)\to\mathrm{Spec}(A)$ and thus the question of
whether the induced map
$\mathrm{Spec}(A_{K^\mathrm{sep}})/\Gamma\to\mathrm{Spec}(A)$ is a
homeomorphism is implied by the claim that
$\mathrm{Spec}(A_L)/\mathrm{Gal}(L/K)\to\mathrm{Spec}(A)$ is a
homeomorphism for all $L$. This then is simple (e.g. it follows
(essentially) from Theorem 1 above).
This also indicates some generality in which your profinite group
question may have an answer. Namely, if $\varprojlim G_i$ is a
projective limit of finite groups acting on $\varprojlim X_i$, where
each $X_i$ is affine, then you probably get what you want. I just now,
embarassingly, have recalled that questions of this form are
considered quite often in Shimura varieties and are discussed (for
instance) in [Deligne] and [Milne].

I’ll think about it more if I have time.
$(\ast\ast)$: The definition of a reductive group is somewhat technical. A reductive group is, in particular, an affine group scheme over $k$ of finite type. In characteristic $0$ one can define it as one all of whose representations are semisimple. In positive characteristic one needs to be more careful. Ask me if you’d like further explanation. Examples of reductive groups are: $\mathrm{GL}_n$, $\mathrm{SL}_n$, $\mathrm{PGL}_n$, $\mathrm{Sp}_{2n}$, any torus, finite constant groups with order coprime to the characteristic of $k$,… Examples of non reductive groups are: $\mathbf{G}_a$, the upper triangular matrices in $\mathrm{GL}_n$,…
References:
[Deligne] Deligne, P., 1979, June. Variétés de Shimura: interprétation modulaire, et techniques de construction de modeles canoniques. In Automorphic forms, representations and L-functions (Proc. Sympos. Pure Math., Oregon State Univ., Corvallis, Ore., 1977), Part (Vol. 2, pp. 247-289).
[GW] Görtz, U. and Wedhorn, T., 2010. Algebraic Geometry I: Schemes. Vieweg+ Teubner.
[Milne] Milne, J.S., 1990. Canonical models of (mixed) Shimura varieties and automorphic vector bundles. Automorphic forms, Shimura varieties, and L-functions, 1, pp.283-414.
[MF] Mumford, D., Fogarty, J. and Kirwan, F., 1994. Geometric invariant theory (Vol. 34). Springer Science & Business Media.
[Scholze] Scholze, Peter. Etale cohomology of diamonds. http://www.math.uni-bonn.de/people/scholze/EtCohDiamonds.pdf
A: Question: "For infinite groups, the quotient as defined above need not be a scheme. But it exists nevertheless as a topological space. In this case, it is still true that the topological space $(SpecA)/G$ is homeomorphic to $Spec(A^G)$?"
Answer: If $H \subseteq G$ are linear algebraic groups over a field $k$, you may construct the quotient $\pi: G \rightarrow G/H$ and $G/H$ is a smooth quasi projective scheme of finite type over $k$.
Remark: "Dear @Con and hm2020, I've edited the post with a better explanation of what I mean."
Example: If $W \subseteq V$ is an $m$-dimensional subspace of and $n$-dimensional $k$-vector space $V$, let $P \subseteq SL(V)$ be the subgroup of elements "fixing" $W$. It follows $\pi: SL(V) \rightarrow SL(V)/P$ is the grassmannian $\mathbb{G}(m,V)$ parametrizing $m$-dimensional subspaces  of $V$. Hence in these cases you always get a quasi projective scheme. In fact the quotient map $\pi$ is locally trivial in the etale topology with fiber $H$ - it is a "principal fiber bundle in the etale topology".
Note: Both groups $P:=Spec(A)$ and $SL(V):=Spec(B)$ are affine schemes of finite type over $k$. There can be no isomorphism $SL(V)/P \cong Spec(B^T)$ for some group $T$ acting on $B$, since $SL(V)/P$ is quasi projective and non-affine in general. When the subgroup $P \subseteq SL(V)$ is parabolic, it follows the quotient $SL(V)/P$ is a projective scheme which is non-affine.
You should take a look at Borel's book "Linear algebraic groups" and Theorem 6.8, page 98. I believe this gives a construction of a quotient G/H for any field k.
@Gabriel - The answer was a "partial answer", meant to indicate that "in some cases you can construct quotients of schemes by "infinite groups" and where the quotient is a scheme". The process of constructing quotients in algebraic geometry is a complicated process - I believe your question on the relation between $Spec(A^G)$ and the quotient topological space $Spec(A)/G$ is answered in an exercise in Atiyah-Macdonald, Chapter 5 on integral extensions. There is also a treatment in Lius book on algebraic geometry.
In the exercise (Ex.5.13) they prove that if $A$ is a commutative unital ring and if $G \subseteq Aut(A)$ is a finite group, it follows for any prime ideal $\mathfrak{q} \subseteq A^G$, the group $G$ acts transitively on the set of prime ideals $\mathfrak{p} \subseteq A$ with $\mathfrak{p}\cap A^G = \mathfrak{q}$.
Question: "Let G be a group acting on a ring A. I would like to know in which generality we know that $Spec(A^G)≅(SpecA)/G$?"
Hence in this case it follows there is a homeomorphism
$$Spec(A)/G \cong Spec(A^G).$$
Any element $g \in G$ is a ring automorphism $g:A \rightarrow A$ and you get for any prime ideal $\mathfrak{p} \subseteq A$ an induced prime ideal $g^{-1}(\mathfrak{p})\subseteq A$. This defines an action of $G$ on $Spec(A)$. You define two prime ideals $\mathfrak{p}, \mathfrak{p}'$ to be equivalent iff there is an element $g\in G$ with $g\mathfrak{p}=\mathfrak{p}'$. This equivalence relation defines the quotient topological space $Spec(A)/G$. And by the exercise: The points in $Spec(A^G)$ are in 1-1 correspondence with orbits of $G$ in $Spec(A)$. Hence there is a homeomorphism $Spec(A)/G \cong Spec(A^G)$
Example: The symmetric group $G$ on two elements acts on $A:=k[x,y]$ and the invariant ring $A^G=k[s_1,s_2]$ is the subring on the elementary symmetric functions $s_1=x+y, s_2=xy$. The sub ring $A^G$ is a polynomial ring in the two functions $s_1,s_2$ - they are "algebraically independent".
As a consequence the quotient map
$$\pi: \mathbb{A}^2_k \rightarrow \mathbb{A}^2_k/G:=Sym^2(\mathbb{A}^1_k) \cong \mathbb{A}^2_k$$
gives an isomorphism between the symmetric product $Sym^2(\mathbb{A}^1_k)$ and affine two space.
Note: In Waterhouse, "Affine group schemes", Ch.16.2 they prove that for any affine group scheme $G$ over a field $k$ and any closed normal subgroup $N \subseteq G$, the quotient $G/N$ exists and $G/N$ is an affine group scheme over $k$. Hence if $G:=SL(V)$, you get for any normal subgroup scheme $N$ a quotient $G/N$ that is an affine scheme, and for a parabolic subgroup scheme $P$ you get a quotient $G/P$ that is a projective scheme. Hence the quotient construction is not simply "taking the Spec of the invariant ring"
$Spec(B^T)$ for some group $T$ acting on $B$. I read somewhere that the study of quotients $\pi: G \rightarrow G/H$ for affine group schemes $H \subseteq G$ and the fact that $\pi$ trivialize in the etale topology (and not in the Zariski topology in general) "inspired" the study of the etale topology and etale descent. Hence if you want to study principal fiber bundles in algebraic geometry/algebra you will need the etale topology.
Why is the projection map $\mathbb{A}^{n+1}_k\setminus \{\vec 0\} \to \mathbb{P}^n_k$ a morphism of schemes?
Quotient of a Lie algebra by a subalgebra - what is it?
