# CCM $3\times3$ into $4\times4$ (image processing)

## Preface

Color Correction Matrix (a.k.a. CCM or CMX) is applied to raw image data (before de-mosaicking, usually from a standard bayer $$RGGB$$ or $$BGGR$$ pattern) to correct for the physical characteristics of the digital image sensor, most importantly for a so called "cross-talk" between sensor elements.

For reference: the most universally used bayer pattern produces pixel values with alternating "color brightness" at pre-assigned positions, like the following picture:

which are then processed to produce "real color" $$RGB$$ triplets per pixel with the assumption that chroma doesn't change significantly from one pixel to another.

The most common form used in pretty much every document I've found online of that CCM is a $$3\times3$$ matrix for input trio $$(R_0, G_0, B_0)$$ of pixel values used to produce a trio of "color corrected" pixel values $$(R, G, B)$$ like the following:

$$\begin{pmatrix} R \\ G \\ B \\ \end{pmatrix} = \begin{bmatrix} A_{00} & A_{01} & A_{02} \\ A_{10} & A_{11} & A_{12} \\ A_{20} & A_{21} & A_{22} \\ \end{bmatrix} \cdot \begin{pmatrix} R_0 \\ G_0 \\ B_0 \\ \end{pmatrix}$$

That assumes that values for Green are averaged out (or some other operation is done to produce 3 values from the original 4).

However the real input data I'm working with would be a stream of true 4-values in $$2\times2$$ blocks $$\begin{vmatrix}B_0&G_{b0}\\G_{r0}&R_0\end{vmatrix}$$.

Quite obviously for me to preserve the values for $$G_b$$ and $$G_r$$ I'd like to perform the color correction using a $$4\times4$$ matrix, not a $$3\times3$$. Note that the order of pixel values is now reversed from $$RGB$$ to $$BGGR$$:

$$\begin{pmatrix} B \\ G_b \\ G_r \\ R \\ \end{pmatrix} = \begin{bmatrix} C_{00} & C_{01} & C_{02} & C_{03} \\ C_{10} & C_{11} & C_{12} & C_{13} \\ C_{20} & C_{21} & C_{22} & C_{23} \\ C_{30} & C_{31} & C_{32} & C_{33} \\ \end{bmatrix} \cdot \begin{pmatrix} B_0 \\ G_{b0} \\ G_{r0} \\ R_0 \\ \end{pmatrix}$$

## Conversion

So here's my struggle: how do I go from that $$3\times3$$ color correction matrix to the final $$4\times4$$ form?

Of course the first thing to do is straight-forward: flip the matrix along the $$A_{20}..A_{02}$$ diagonal and then... add a row and a column?

The Green is the one that gets "split" into two independent values, one for each row (blue and red). So my thinking is to just blindly do exactly that:

$$C = \begin{bmatrix} A_{22} & A_{12} & A_{12} & A_{02} \\ A_{21} & A_{11} & A_{11} & A_{01} \\ A_{21} & A_{11} & A_{11} & A_{01} \\ A_{20} & A_{10} & A_{10} & A_{00} \\ \end{bmatrix}$$

## Question

Would like to know if the above is the correct way to do such conversion? A bit of explanation beyond a "yes" or "no" would be most welcome, too.

After extensive testing and some more investigation and research it became clear that this would not be the correct way to do the color correction (as stated) in a first place. Primarily due to the fact that an important dependency is missing in the proposed approach: namely the fact that $$RGB$$ trio depends on the specifics of the de-mosaicing.