Trying to use Ray Casting with lumaSplats but the Ray does not hit anything

[Limonka11]

I am trying to get the 3D position of a mouse click inside of the LumaSplats mesh.

This is part of the code that I have written:

  function getClicked3DPoint(evt: any) {
    // evt.preventDefault();
    const canvasPosition = canvasRef.current?.getBoundingClientRect();
    console.log('WTF', scene.children)

    mousePosition.x = ((evt.clientX - canvasPosition!.left) / canvasPosition!.width) * 2 - 1;
    mousePosition.y = -((evt.clientY - canvasPosition!.top) / canvasPosition!.height) * 2 + 1;

    rayCaster.setFromCamera(mousePosition, camera); //cameraRef.current!);
    let intersects = rayCaster.intersectObjects(scene.children, true);

    if (intersects.length > 0)
        console.log(intersects[0].point)
        return intersects[0].point;
  }

      <Canvas
        ref={canvasRef}
        frameloop={frameLoop}
        gl={{
          antialias: false,
        }}
        camera={{
          position: [0, 0, 0],
        }}
      >
          <group
            dispose={null}>
            <lumaSplats
              onPointerMissed={() => console.log("you clicked, but hit nothing at all")}
              onClick={(e) => {
                console.log(e.point) // this is the coordinates for the event, eg. mouse click
                getClicked3DPoint(e)
              }}
              onLoad={() => {
                setIsLoading(false);

                if (camera360Ref.current) {
                  camera360Ref.current.object.position.x = -5;
                  camera360Ref.current.object.position.y = 0.4;
                  camera360Ref.current.object.position.z = 7;
                }
              }}
              loadingAnimationEnabled={true}
              semanticsMask={LumaSplatsSemantics.ALL}
              // onBeforeRender={}
              enableThreeShaderIntegration={false}
              particleRevealEnabled={true}
              source={src}
              position={[0, 0, 0]}
              scale={5}
              rotation={[0, 0, 0]}
            />

            <CustomLine start={[0,0,0]} end={[1,1,0]} />
          </group>
     </Canvas>

I have created the following luma capture: https://lumalabs.ai/capture/71f64429-3872-417a-a67c-a8b6c6ab16d3

As you can see it is a capture of a normal room. What I want to achieve is to draw lines on the splat. For this, I want to click on the canvas, this will cast a ray to the first intersecting part of the splat. I will take the coordinates of this intersection and then by clicking on another part of the splat I want to draw a line between these two points.

However, most of the times the event on onPointerMissed is being activated and thus no intersecting point is found. Do you have any idea why this is happening and what I can do in order to fix this?

[abrahamezzeddine]

Try intersecting the point cloud itself rather than the splat and see how that works.

[Limonka11]

Try intersecting the point cloud itself rather than the splat and see how that works.

Hi @abrahamezzeddine, How can I get the point cloud?

[abrahamezzeddine]

Each Gaussian splat file contains XYZ coordinates for the center coordinates of the splat. Essentially, a splat file is a point cloud where each point is stretched in different directions.

You do not need to render the point cloud, it is enough that you try to “hit” the XYZ.

Perhaps you also need to take into consideration that the points are too small for your clicks. Perhaps increase size of each point.

[Limonka11]

Yes, I am indeed trying to hit the XYZ coordinates of a point using ray-casting

This is what this function is supposed to do:
` function getClicked3DPoint(evt: any) {
// evt.preventDefault();
const canvasPosition = canvasRef.current?.getBoundingClientRect();
console.log('WTF', scene.children)

mousePosition.x = ((evt.clientX - canvasPosition!.left) / canvasPosition!.width) * 2 - 1;
mousePosition.y = -((evt.clientY - canvasPosition!.top) / canvasPosition!.height) * 2 + 1;

rayCaster.setFromCamera(mousePosition, camera); //cameraRef.current!);
let intersects = rayCaster.intersectObjects(scene.children, true);

if (intersects.length > 0)
    console.log(intersects[0].point)
    return intersects[0].point;

}`

However, most of the time it doesn't hit anything and sometimes it hits something that appears to be in the centre of the room in the air. I thought that maybe Luma's software is suitable only for objects and tries to represent the room as an object in the centre, not as an open space.

Do you have any suggestions? Does the function look correct to you?

[abrahamezzeddine]

I do not know exactly how lunas software works, I just happened to pass by here.

I tried this with Mkkellogs splat viewer. I first tried to intersect the splat itself which did not work (since splats are “invisible” in three.js when raycasting) and then I had to try with the xyz points which did not work so good.

I had to increase the size of the point, xyz only, in order to be able to raytrace the points. I do not have that code at hand at the moment however.

It can be that your points are very very small that you will not able to hit them. That is why I suggested you try to increase the point size, or try to load the point cloud concurrently, only fetching the xyz since that is what you need.