Spherical harmonics parametrization

[VladimirYugay]

In the original splatting repo, spherical harmonics parameters are represented as features_rest of shape (n, 15, 3) and features_dc of shape (n, 1, 3) where n is the number of points.

Here the feature vector is of shape 56 by default. First 4 dimensions for rotation, then 3 dimensions for the scale, 1 dimensions for alpha. This leaves 48 dimensions for the spherical harmonics parameters.

How can I convert the optimized features from the original repo to match the features that you have preserving the correct order?

[jb-ye]

I'm afraid matching the order of features isn't enough. I found that the official repo was computing color by max(0, <SH, dir> + 0.5), while this unofficial code compute color using sigmoid(<SH, dir>).

I also have this question for the @wanmeihuali : what is the design decision to use sigmoid?

[wanmeihuali]

Hi @jb-ye, you are right. there are some differences between the official repo and this repo. Consider the volume rendering formula:

$$ C = \sum_{i=0}^{N-1} \alpha_i c_i \prod_{j=0}^{i-1} (1 - \alpha_j) $$

$c_i$ is the color of each Gaussian, and in most NeRF implementations, its range is $[0, 1]$. When I implemented this repo, because the official repo had not been released, I followed the common practice and used sigmoid to map the range of $c_i$ to $[0, 1]$. However, the official repo uses $max(0, c_i + 0.5)$ to map the range of $c_i$ to $[0, \infty]$, which cause possible overflow in the volume rendering formula, and they clip the pixel color to $[0, 1]$ after the volume rendering formula. I've tried to switch to the official repo's color mapping method, but the result gets worse, so I keep the sigmoid method. Switching between these two methods shall be easy(less than 20 lines of code).

@VladimirYugay Please see

taichi_3d_gaussian_splatting/benchmark/inference_benchmark.py

Line 65 in 2cea5de

point_cloud = torch.from_numpy(xyz).float()

for how to import model trained by official implementation, Also you need to modify the color formula a bit.

[jb-ye]

@wanmeihuali Thanks for the confirmation. I also tried to switch from sigmoid to ReLU and observed the performance degradation. Any idea why it happens? What makes sigmoid more favorable for optimization/training?

I have been trying to find a way to either optimize the color mapping as the official repo did yet don't compromising on the quality or post-process SH to reflect the original color mapping.

I have two reasons for doing that:

(1) many people will be interested to compare taichi gs to official gs in an interactive viewer (e.g. https://antimatter15.com/splat/ )

(2) It was observed that storing spherical harmonics and quantize them in low bit representation can greatly reduce the file size of GS models. However, with sigmoid mapping, it is not possible to quantize SH.

[oliver-batchelor]

I also have been trying to load point clouds from the original repo and try them with the Taichi version.
I found two things - the adjusted color calculation (below), and the quaternion order was different.

One thing I haven't been able to figure out is some weird blocking artefacts, they look like specific Gaussians are causing some kind of overflow/NaN or something in the surrounding tiles. I checked and there's no inf or nan in the input data, so I'm not sure what's causing that.

Figured this out: (log) scale feature which was almost zero in a small number of points. Clamping this at some small value and it's pretty much identical to the original.

One thing to note - in the original implementation often times there seem to be large distracting Gaussians in the background which sort to the foreground unexpected, in the Taichi version this doesn't seem to happen and is overall better.

@ti.func
    def get_color_by_ray(
        self,
        ray_origin: ti.math.vec3,
        ray_direction: ti.math.vec3,
    ) -> ti.math.vec3:
        o = ray_origin
        d = ray_direction
        # TODO: try other methods to get the query point for SH, e.g. the intersection point of the ray and the ellipsoid
        
        rgb = ti.math.vec3(*[SphericalHarmonics(c).evaluate(d) 
                              for c in [self.color_r, self.color_g, self.color_b]])
        return ti.math.clamp(rgb + 0.5, 0, 1)
            # return ti_sigmoid(rgb)

[jb-ye]

@oliver-batchelor @VladimirYugay Latest commit, you can find sample code to order SH to match the one used by official GS. https://github.com/wanmeihuali/taichi_3d_gaussian_splatting/blob/main/taichi_3d_gaussian_splatting/GaussianPointCloudScene.py#L148

[VladimirYugay]

@wanmeihuali Thanks, that script is helpful!

Can you elaborate on Also you need to modify the color formula a bit?

I was able to load all the parameters and render the image. However, my image looks pale (colors are not so bright), and I think that's because I need to modify the color formula somehow.

[oliver-batchelor]

@wanmeihuali Thanks, that script is helpful!

Can you elaborate on Also you need to modify the color formula a bit?

I was able to load all the parameters and render the image. However, my image looks pale (colors are not so bright), and I think that's because I need to modify the color formula somehow.

See the code above - the difference is

taichi version:
sigmoid(sh(dir, params))

3dgs original:
max(0, 0.5 + sh(dir, params))

[VladimirYugay]

@oliver-batchelor, but I'd also need to change the backward pass if I want to optimize the taichi representation with changed activation functions. Should I change ti_sigmoid_with_jacobian to ti_relu_with_jacobian or is there an easier way to achieve this?

[oliver-batchelor]

Yeah, you would. I have only tried displaying them at the moment.

…

On Sat, 28 Oct 2023 at 02:05, Vladimir Yugay ***@***.***> wrote: @oliver-batchelor <https://github.com/oliver-batchelor>, but I'd also need to change the backward pass if I want to optimize the taichi representation with changed activation functions. Or am I missing something? — Reply to this email directly, view it on GitHub <#150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAITRZIA7HNMNXN7TRWIJW3YBOWRLAVCNFSM6AAAAAA6JMB44OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBSHA4DINRRGA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[VladimirYugay]

@wanmeihuali I'm not familiar with taichi, and I was wondering whether adding this will also require the change of the gradient for the backward pass:

  r = SphericalHarmonics(self.color_r).evaluate(d)
  r = - ti.math.log(1 / (ti.math.clamp(r + 0.5, 0, 1)) - 1)
  r_normalized = ti_sigmoid(r)

I found that with this transformation the features from the original report become compatible with the ones in taichi

[wanmeihuali]

@wanmeihuali I'm not familiar with taichi, and I was wondering whether adding this will also require the change of the gradient for the backward pass:

  r = SphericalHarmonics(self.color_r).evaluate(d)
  r = - ti.math.log(1 / (ti.math.clamp(r + 0.5, 0, 1)) - 1)
  r_normalized = ti_sigmoid(r)

I found that with this transformation the features from the original report become compatible with the ones in taichi

@VladimirYugay For backward, it should be here:

taichi_3d_gaussian_splatting/taichi_3d_gaussian_splatting/GaussianPoint3D.py

Lines 361 to 370 in 2447148

	r_normalized, r_normalized_jacobian = ti_sigmoid_with_jacobian(r)
	g, g_jacobian = SphericalHarmonics(
	self.color_g).evaluate_with_jacobian(d)
	g_normalized, g_normalized_jacobian = ti_sigmoid_with_jacobian(g)
	b, b_jacobian = SphericalHarmonics(
	self.color_b).evaluate_with_jacobian(d)
	b_normalized, b_normalized_jacobian = ti_sigmoid_with_jacobian(b)
	r_jacobian = r_normalized_jacobian * r_jacobian
	g_jacobian = g_normalized_jacobian * g_jacobian
	b_jacobian = b_normalized_jacobian * b_jacobian

if we remove sigmoid and uses a relu(x+0.5) in stead, then the jocabian shall be the original value when x>0 and 0 then x <=0