Back to blog

Masking: click one object, get a 3D scene of just that

Splat3D team··2 min read
featuremaskingapi

Today I'm shipping Masking: click one object in your capture, and Splat3D trains a splat of just that object, with no background.

How it works

You click the object in one frame of a video, or in one photo of a zip. Splat3D finds it in every other frame and trains only those pixels.

Under the hood there are three steps:

  1. Camera poses are solved on the whole frame first, because the background is what places the camera.
  2. SAM 2.1 tracks the object through the frames. For a zip, the photos are ordered by camera position first.
  3. The splat trains on the masked pixels only.

How to use it

On splat-3d.com/upload, turn on the Masking toggle, then click Mask this object.

In the API it's one optional subject field when you create a scene. For a video, give the moment and your clicks:

{
  "content_type": "video/mp4",
  "subject": { "time_s": 12.4, "points": [[0.52, 0.61], [0.50, 0.70]] }
}

For a zip of photos, name the photo instead:

{
  "content_type": "application/zip",
  "subject": { "image": "trip/IMG_0042.JPG", "points": [[0.52, 0.61]] }
}

You can send 1 to 5 clicks, each a fraction of the picture as displayed. The full rules are in the Masking guide.

A masked scene costs the same as a normal one.

What it doesn't do well yet

  • Anything that passes in front of the object gets carved out of it.
  • A subject that moves during the capture smears.
  • Very thin edges come out wispy.

Try it