This problem has been eating at me for a little while now, and it's extremely frustrating.

First off, let me begin with explaining the matrices I am using:

Matrix breakdown

Unity calculates culling matrices incorrectly. From the matrix, it seems to generate the actual occlusion culling itself scaled proportionally to the projection matrix, at least when it comes to the near clipping plane. This can typically be fixed by simply calculating a new projection matrix with half the near clip plane distance, but my problem is unfortunately not that simple. I want to calculate an oblique culling matrix (that is, with the near plane not perpendicular to the camera), and this near-clip-doubling is making it impossible to do as no matter what method I use to calculate the matrix, it is proportionally incorrect. I cannot figure out how to counter-compute against the scaling being caused by Unity's incorrect calculation, as I am simply not familiar enough with linear algebra and matrix math to intuitively understand the problem.

Here is the code that is being used to calculate the oblique matrix

//clipPlane is a vector4 in camera space (direction + distance from camera)
Matrix4x4 CalculateObliqueOcclusion(Matrix4x4 projection, Vector4 clipPlane, float signSide) {
    //signSide determines whether it is the near or the far clipping plane that moves to the clipPlane
    Vector4 q = projection.inverse * new Vector4(
    Vector4 c = clipPlane * (2F / (Vector4.Dot(clipPlane, q)));
    //Left/Right sizing
    projection[2] = c.x - projection[3];
    //Top/Down sizing
    projection[6] = c.y - projection[7];
    //Near/Far clipping plane adjustment. Used to move far clipping plane.
    projection[10] = (c.z - projection[11]);
    //Scale adjustment- used to move near clipping plane
    projection[14] = (c.w - projection[15]);
    return projection;

Here is a video demonstrating the issue. the clipping plane is being generated correctly off of the object, but as you can see, the culling matrix is wildly out of control.

Occlusion matrix being calculated incorrectly

In this example, the invisible "box" that I am grabbing is being used to calculate the near plane of the projection matrix to be used — this code is working properly. The green box is a rendering of the volume of the occlusion culling matrix, which as you can see appears to scale with the rear plane by a factor of two.

  • $\begingroup$ You might explain what the variables in the matrix breakdown mean instead of forcing everyone to guess. $\endgroup$
    – amd
    Jan 17 '17 at 20:23
  • $\begingroup$ @amd Sorry about that! It only let me post one picture and I didn't have access to anything to edit images with. Here's the breakdown + visual representation. i.imgur.com/48xm3l3.png $\endgroup$
    – Phygon
    Jan 18 '17 at 0:39
  • $\begingroup$ Looks like it uses the OpenGL conventions for the occlusion matrix. Do you understand how it’s derived in the first place? It’s not simply a rescaling of the projection matrix. The $x$- and $y$- parts are also modified so that the frustrum’s sides are “straightened out” and it gets mapped into a cube. $\endgroup$
    – amd
    Jan 18 '17 at 5:04
  • $\begingroup$ So, if I’m understanding you correctly, you’re trying to build a perspective transformation that will map a frustrum with an oblique cut (instead of the usual orthogonal cut) into the cube with vertices at $(\pm1,\pm1)$. Is that about right? $\endgroup$
    – amd
    Jan 18 '17 at 5:07
  • $\begingroup$ @amd What I'm attempting to do is modify an existing projection matrix by moving its near clipping plane to an arbitrary plane in camera space. The cube is merely the reference point being used to generate the plane. Here is an image example, since it might be more effective: i.imgur.com/JRYs4r7.png. This matrix is actually being generated correctly for the camera's projection matrix. The green matrix shown in the video, the culling matrix, is calculated incorrectly by Unity. I am attempting to correct for this, but I am not sure how I could. Also, thanks a ton for trying to help me with this. $\endgroup$
    – Phygon
    Jan 18 '17 at 8:30

It might not be possible to fix your problem completely, but I’ll point you at something to try.

Issues with the view volume are inherent to the method that you’re using to get an oblique near plane. A paper by Eric Lengyel covers this in detail. Your code snippet uses the results described in that paper. The basic issue is that the near and far planes are coupled, so changing the near plane affects the far plane in an undesirable way. The far plane can be adjusted to some extent, but not enough to map the view volume onto the entire standardized cube. This exacerbates the floating-point resolution issues that the projection matrix already has.

I’ll assume the same OpenGL conventions for the projection matrix that Unity and Lengyel use. In homogeneous coordinates, a plane in $\mathbb{RP}^3$ can be represented by a (covariant) vector normal to the corresponding subspace in $\mathbb R^4$. The projection matrix $\mathbf P$ maps a view volume, which is the frustrum of a pyramid, onto the cube with vertices at $(\pm1,\pm1,\pm1)$. It turns out that the bounding planes of the view volume can be recovered as sums and differences of the first three rows of $\mathbf P$ with the last one. In particular, the near plane is $P_3+P_4$ and the far plane is $P_3-P_4$. So, one can force the near plane to to be a given plane $C$ by changing the third row of $\mathbf P$ to $C-P_4$. However, doing so also changes the far plane to $C-2P_4$. The fourth row can’t be changed without affecting all of the other planes, but $C$ can be multiplied by a scale factor without changing the plane that it represents, so the third row instead gets set to $\alpha C-P_4$, where $\alpha$ is chosen to place the far plane into a better position. Unfortunately, the range of pseudodepth values that gets used is reduced in the process. As Lengyel shows, the range gets smaller with increasing tilt of and distance to the oblique plane. The standard frustrum with near/far planes perpendicular to the view axis already has potential problems with occlusion computations due to floating-point resolution, and reducing the overall range of values certainly doesn’t improve matters.

The above is certainly going to be an issue, but I don’t think that it explains the gross failures shown in your video. One thing to look into is the value of q, which corresponds to the point $Q$ in Lengyel’s paper. This is supposed to be the vertex of the original frustrum that’s opposite $C$ and is used to position the far plane. In the paper, $Q$ is computed by mapping $C$ into the standard view volume to get the plane $C'$ and then mapping the corner $Q'$ of the standard cube that’s opposite this plane back into camera space. The paper assumes that the signs of the first two components of $C'$ match those of $C$, which for most perspective transformations is a pretty safe assumption. Your code snippet further assumes that the signs of both components are the same, so it always chooses either $(1,1,1)$ or $(-1,-1,1)$ for $Q'$, which might not actually be the corner opposite to $C'$. For some tilts of $C$ relative to the view axis, this will misplace the far plane and in some cases might cause it to intersect the field of view behind the camera. That kind of looks like what’s going on in the obvious failures. If that’s really the problem, fixing it could be as simple as changing the first two components of the vector that’s back-mapped to q to be signSide times the signs of c.x and c.y, respectively.

Postscript: A simple way to describe the coupling of the pairs of opposite faces of the view frustrum is that their normals and the direction vector of the view axis must be coplanar.

  • $\begingroup$ Thank you so much for your help! Your explanation gives me a much clearer understanding of the possible causes of my problem than I had before, and I will hopefully be able to fix the problem from the info contained within your answer. The far-plane values are the ones that I was unable to correctly scale, so your solution will likely work. Floating point issues shouldn't be too much of a problem as z-fighting isn't a problem with a non-visual matrix. If I can crack the problem, I'll update the original post and share it with the Unity community as this is a pervasive issue. Again, thank you! $\endgroup$
    – Phygon
    Jan 19 '17 at 22:11
  • $\begingroup$ @Phygon Good luck! It’s been a long while since I’ve done any third-person debugging :) $\endgroup$
    – amd
    Jan 19 '17 at 22:13
  • $\begingroup$ Well, your comment, explanation, and everything was incredibly helpful, but it looks like this just isn't an egg I can crack. Due to the particular nature of Unity's bizarrely incorrect scaling (as it only moves the near clipping plane incorrectly, seen in i.imgur.com/kI65Qsn.png and not the matrix values themselves), I can't seem to implement a functioning workaround as any value of α and any modification of q can prevent the occlusion matrix from overflowing from the projection matrix, but it cannot make it equal (or even roughly equal). Damn, that really sucks. $\endgroup$
    – Phygon
    Jan 20 '17 at 21:42
  • $\begingroup$ @Phygon That’s too bad. I’m going to maintain that there’s a bug in CalculateObliqueOcclusion above that gets q wrong 75% of the time, but unless the oblique plane has an extreme tilt, it likely doesn’t make much difference. It sounds like there are other things going on as well. The first thing I would ask myself is why are the projection and occlusion matrices different in the first place? That doesn’t entirely make sense to me. Good luck! $\endgroup$
    – amd
    Jan 21 '17 at 19:49
  • $\begingroup$ Well, that was the bug that I was attempting to work around. Unity just.. doesn't calculate them correctly, and they probably never realized since near clipping planes are typically so close to the camera. Thankfully, I opened a bug report and they said they were able to reproduce and isolate the problem, so it is likely going to be fixed :). Also, You are correct about the q computation, and modifying the code to account for it gets a more reliable (albeit still incorrect due to the unity bug) result. Thank you so much for your help! It means a lot. $\endgroup$
    – Phygon
    Jan 22 '17 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.