Back to blog
Masking: click one object, get a 3D scene of just that
Splat3D team··2 min read
featuremaskingapi
Today I'm shipping Masking: click one object in your capture, and Splat3D trains a splat of just that object, with no background.
How it works
You click the object in one frame of a video, or in one photo of a zip. Splat3D finds it in every other frame and trains only those pixels.
Under the hood there are three steps:
- Camera poses are solved on the whole frame first, because the background is what places the camera.
- SAM 2.1 tracks the object through the frames. For a zip, the photos are ordered by camera position first.
- The splat trains on the masked pixels only.
How to use it
On splat-3d.com/upload, turn on the Masking toggle, then click Mask this object.
In the API it's one optional subject field when you create a scene. For a video, give the moment and your clicks:
{
"content_type": "video/mp4",
"subject": { "time_s": 12.4, "points": [[0.52, 0.61], [0.50, 0.70]] }
}
For a zip of photos, name the photo instead:
{
"content_type": "application/zip",
"subject": { "image": "trip/IMG_0042.JPG", "points": [[0.52, 0.61]] }
}
You can send 1 to 5 clicks, each a fraction of the picture as displayed. The full rules are in the Masking guide.
A masked scene costs the same as a normal one.
What it doesn't do well yet
- Anything that passes in front of the object gets carved out of it.
- A subject that moves during the capture smears.
- Very thin edges come out wispy.
Try it
- Mask your first object: splat-3d.com/upload
- Build with it: Masking guide
- Tell me how it went: Discord. I'm usually around as @zeke.