# How to reduce credible sets in over-specified linear regression while maintaining global coverage probability?

Vectors $$A, B$$ and covariance matrix $$C$$ are fixed and known. I have a vector of measurements, $$Y\in\mathbb{R}^n$$, sampled from $$M_1: Y \sim N(A\alpha_* + B\beta_*, C)$$

My goal, roughly speaking, is to find a small 50% Bayesian credible set, $$S$$, for $$\alpha$$ that is robust: $$S$$ should exhibit at least 50% coverage probability for all value of $$\alpha$$, meaning that if $$Y\sim N(A\alpha_* + B\beta_*, C)$$ for fixed $$(\alpha_*, \beta_*)$$, then $$\alpha_* \in S$$ with probability at least 50%.

The classical approach of jointly estimating $$(\alpha, \beta)$$ yields unacceptably large $$S$$ when $$\beta=0$$, since in this case we are estimating more degrees of freedom than are necessary. If we knew a priori that $$\beta=0$$, we would estimate compute $$S$$ by classical regression on $$Y\sim N(A\alpha, C)$$ and be done. Of course when $$\beta \neq 0$$, this approach results in degraded coverage probability for $$S$$.

I'm looking for a way to get smaller credible sets than classical methods afford when $$\beta=0$$ (or near zero) while maintaining coverage probability for all $$\alpha_*, \beta_*$$.

Here is one idea. Consider the Bayesian posterior distribution of $$\alpha$$ assuming flat (improper) prior distributions on $$\alpha, \beta$$: $$P(\alpha | Y) \propto \int P(Y|\alpha, \beta)\, d\beta$$ where the likelihood function $$P(Y|\alpha, \beta)$$ is given by the normal distribution above. The posterior $$P(\alpha|Y)$$ is Gaussian with mean and variance determined by classical linear regression. Instead, introduce an informative prior $$\pi(\beta)\propto P(Y|\beta)$$, where $$P(Y|\beta):=\int P(Y|\alpha, \beta)\, d\alpha$$ so that $$P(\alpha|Y) \propto \int P(Y|\alpha, \beta) P(Y|\beta) \, d\beta$$ In cases were $$\beta$$ is near 0, the integrand should be more concentrated near the mode of the likelihood function, thus reducing the size of $$S$$. When $$\beta \neq 0$$ and is 'observable', $$P(Y|\beta)$$ should be concentrated near the correct value.

Using the same data in the likelihood and prior in this way feels abusive, but it's all I've come up with so far.