Question about proof verifiers like metamath. How does metamath or other proof verifiers determine if two propositional formulas can be made equal?  Pointers to the literature would be appreciated.
 A: Before I begin, I want to point out that most proof verifiers are very much unlike Metamath, so my answer may not apply well to the way this sort of thing is dealt with by other verifiers.
There are a few slightly different things you may be wondering about, and most (but not all) might be answered by the Metamath book.
1. What if you have a proof to verify?
If you have a complete proof in the Metamath language (that any verifier can check), then this is a non-issue, because the proof must specify (in an indirect way that is specific to Metamath) what gets substituted for each variable.
2. How do programs know without a complete proof?
Programs related to Metamath (e.g. "the Metamath program" and "mmj2") wouldn't need a complete proof to figure something like this out (or at least to narrow down the possibilities). I haven't looked through the source of either program, but I can describe one way you could do it, at least for the simple example included in your comment.
Since parentheses and -> are traditionally constants that cannot be replaced (substituted into), if your axiom is P -> ( Q -> P ), we can usually figure things out by matching up those constants. For instance, if no formula has unpaired parentheses, that would help a lot.
Even in a contrived case like -> ( -> ( -> ( -> -> ( ) we know that P must end in ( by looking at the end of the string, so from the beginning we know it's either -> ( -> ( or -> ( (as otherwise we wouldn't have the ( before Q from the axiom). But the end then shows that it can't be -> ( -> (, so P must be -> (. That means this reduces to P -> ( -> ( -> P ). But then matching up the beginning and end shows that Q must also be -> (.
In a weirder case, there could be multiple options (e.g. if your axiom were P -> Q), but the way we usually set math up makes that rare (impossible?).

Bonus: Metamath code
For those unfamiliar with metamath, I have built an example demonstrating what I was talking about in point 2.
Let's say we have the below metamath source, which declares:

*

*constant symbols that can't be substituted into

*variables that can be substituted into

*the types of the things that can be substituted into each variable (they all take formulas)

*an axiom (named "impf") stating that things of the form ( P -> Q ) are formulas

*an axiom (named "ax") stating that things of the form P -> (Q -> P) are true.

*a claimed provable theorem that A -> ( ( A -> A ) -> A ) is true, along with a partial proof that involves two unknown steps and then ends with an invocation of ax.

Then the metamath program knows that since ax involves the variables P and Q, which stand for formulas, the two steps better be
$c ( $.
$c ) $.
$c -> $.
$c formula $.
$c true $.
$v P $.
$v Q $.
$v A $.
pf $f formula P $.
qf $f formula Q $.
af $f formula A $.
impf $a formula ( P -> Q ) $.
ax $a true P -> ( Q -> P ) $.
thm $p true A -> ( ( A -> A ) -> A ) $= ? ? ax $.

If you load the above code (with a line-break at the end) into the metamath program, and then use the commands prove thm and show new_proof /all, it show that it knows you need to show that A and ( A -> A ) will need to be substituted in for the variables in the axiom, in that order (to satisfy pf and qf). Specifically, it displays:
1   pf=?   $? formula A
2   qf=?   $? formula ( A -> A )
3 thm=ax $a true A -> ( ( A -> A ) -> A )

The line of code to include the whole proof instead of the proof with unknown steps is thm $p true A -> ( ( A -> A ) -> A ) $= af af af impf ax $., where the end uses RPN to tell you how to build the proof. The af af impf in the middle will become the needed formula ( A -> A ) when processed.
