feat: Add SAM 3D Objects support for 3D reconstruction

[giswqs]

Summary

Adds support for SAM 3D Objects, Meta's foundation model for reconstructing 3D objects from 2D images using segmentation masks.

Closes #461

Features

Sam3DReconstructor Class

High-level interface for 3D reconstruction:

from samgeo.sam3d import Sam3DReconstructor

reconstructor = Sam3DReconstructor()
output = reconstructor.reconstruct('image.png', 'mask.png')
reconstructor.save_ply(output, 'object.ply')

Integration with SamGeo

Reconstruct 3D models from SamGeo segmentation results:

from samgeo.sam3d import reconstruct_from_samgeo
import geopandas as gpd

masks = gpd.read_file('masks.gpkg')
ply_files = reconstruct_from_samgeo(
    samgeo_result=masks,
    image_path='image.tif',
    output_dir='./3d_models',
)

Helper Functions

• print_install_instructions() - Detailed setup instructions
• reconstruct_multiple() - Batch reconstruction

Requirements

SAM 3D Objects has significant requirements:

• Linux 64-bit system
• NVIDIA GPU with at least 32GB VRAM
• HuggingFace authentication for checkpoint access
• Separate conda environment (see installation instructions)

Documentation

• docs/examples/sam3d.ipynb - Example notebook
• docs/sam3d.md - API reference

Reference

SAM 3D Team (2025). SAM 3D: 3Dfy Anything in Images. https://arxiv.org/abs/2511.16624

[github-actions]

🚀 Deployed on https://69856795fb075d4aa462f8ca--opengeos.netlify.app

[Copilot]

Pull request overview

This PR adds support for SAM 3D Objects, Meta's foundation model for reconstructing 3D objects from 2D images using segmentation masks. The integration provides a high-level interface for converting SamGeo segmentation results into 3D models (Gaussian splats/meshes), enabling a complete workflow from 2D image segmentation to 3D reconstruction.

Changes:

• Added sam3d.py module with Sam3DReconstructor class for 3D reconstruction from masked objects
• Integrated sam3d exports into the main package namespace via __init__.py
• Added documentation and example notebook demonstrating usage

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
samgeo/sam3d.py	Core module implementing SAM 3D Objects integration with image/mask loading, 3D reconstruction, and batch processing capabilities
samgeo/init.py	Exports sam3d functionality to main package namespace for easier imports
mkdocs.yml	Adds sam3d documentation and example notebook to documentation navigation
docs/sam3d.md	API reference documentation page for sam3d module
docs/examples/sam3d.ipynb	Example notebook demonstrating sam3d usage workflows

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The function accesses samgeo_result.iloc[i].geometry without checking if the GeoDataFrame is empty. If samgeo_result has length 0, num_objects will be 0 and the loop won't execute, which is fine. However, if samgeo_result contains rows with None or invalid geometries, this could raise an AttributeError. Consider adding validation to check for None geometries or catch potential AttributeErrors within the try-except block.

Suggested change

	geom = samgeo_result.iloc[i].geometry
	# Rasterize the geometry to create a mask
	mask = rasterize(
	[(geom, 1)],
	out_shape=shape,
	transform=transform,
	fill=0,
	dtype=np.uint8,
	)
	mask = (mask > 0).astype(np.uint8) * 255
	try:
	output = reconstructor.reconstruct(image_path, mask, seed=seed + i)
	output_path = os.path.join(output_dir, f"object_{i:03d}.ply")
	reconstructor.save_ply(output, output_path)
	output_files.append(output_path)
	try:
	geom = getattr(samgeo_result.iloc[i], "geometry", None)
	if geom is None:
	logger.warning(f"Skipping object {i}: geometry is None")
	continue
	# Rasterize the geometry to create a mask
	mask = rasterize(
	[(geom, 1)],
	out_shape=shape,
	transform=transform,
	fill=0,
	dtype=np.uint8,
	)
	mask = (mask > 0).astype(np.uint8) * 255
	output = reconstructor.reconstruct(image_path, mask, seed=seed + i)
	output_path = os.path.join(output_dir, f"object_{i:03d}.ply")
	reconstructor.save_ply(output, output_path)
	output_files.append(output_path)
	except AttributeError as e:
	logger.warning(f"Skipping object {i} due to invalid geometry: {e}")
	continue

[Copilot]

The load_single_mask function is being called with os.path.dirname(mask), which extracts the directory from the mask file path. However, this approach has a critical issue: when a mask path like "mask.png" (without a directory) is provided, os.path.dirname will return an empty string, which would likely cause load_single_mask to fail. Additionally, the index=0 parameter suggests load_single_mask expects to load from multiple masks in a directory, but the user is providing a specific file path. This API design mismatch could cause runtime errors. Consider using a different loading approach that directly reads the mask file, or add validation to ensure the path has a directory component.

Suggested change

	mask = load_single_mask(os.path.dirname(mask), index=0)
	# If a directory is provided, use load_single_mask (original behavior)
	if os.path.isdir(mask):
	mask = load_single_mask(mask, index=0)
	else:
	# Otherwise, treat it as a direct path to a mask image file
	from PIL import Image
	mask = Image.open(mask)

[Copilot]

The type hint indicates that mask can be a PIL.Image.Image object, but there is no handling for this case in the code. The if-elif chain only handles str (lines 208-209) and np.ndarray (lines 210-216), meaning a PIL.Image.Image would be passed through without any processing. This could either work correctly if the inference expects PIL images directly, or fail at runtime. Either add explicit handling for the PIL.Image case or update the type hint to remove it if it's not supported.

[Copilot]

The Path import is imported but never used anywhere in the module. Consider removing it to keep the imports clean.

Suggested change

from pathlib import Path

[Copilot]

The _check_sam3d function is defined but never called anywhere in the module. It appears to be a helper function intended to verify SAM 3D Objects installation, but the Sam3DReconstructor class and other functions don't use it. Consider either using this function in the appropriate places (e.g., in Sam3DReconstructor.init) or removing it if it's unnecessary.

Suggested change

	def _check_sam3d():
	"""Check if SAM 3D Objects is installed and raise informative error if not."""
	try:
	# Try to import from the sam-3d-objects package
	import sys
	# Check if the sam-3d-objects notebook inference module is available
	# The package doesn't have a standard import, so we check for the inference module
	sam3d_path = os.environ.get("SAM3D_PATH")
	if sam3d_path:
	sys.path.insert(0, os.path.join(sam3d_path, "notebook"))
	from inference import Inference
	return Inference
	except ImportError:
	raise ImportError(
	"SAM 3D Objects is not installed or not configured properly.\n\n"
	f"{SAM3D_INSTALL_INSTRUCTIONS}\n\n"
	"After installation, set the SAM3D_PATH environment variable:\n"
	" export SAM3D_PATH=/path/to/sam-3d-objects\n"
	)

[Copilot]

The Tuple import is imported but never used anywhere in the module. Consider removing it to keep the imports clean.

Suggested change

	from typing import Any, Dict, List, Optional, Tuple, Union
	from typing import Any, Dict, List, Optional, Union

[Copilot]

The arXiv ID format "2511.16624" appears suspicious. Standard arXiv IDs after April 2007 follow the format YYMM.NNNNN (e.g., 2501.12345 for January 2025). The ID "2511.16624" would represent November 2025, which is in the future. Additionally, the reference states "SAM 3D Team (2025)" but we are currently in February 2026, and arXiv papers from November 2025 should already be published. Please verify this arXiv ID is correct, as it may not exist or may be a placeholder.

Suggested change

	SAM 3D Team (2025). SAM 3D: 3Dfy Anything in Images.
	https://arxiv.org/abs/2511.16624
	Repository: https://github.com/facebookresearch/sam-3d-objects
	Website: https://ai.meta.com/sam3d/
	"""
	SAM 3D Team. SAM 3D: 3Dfy Anything in Images (arXiv preprint).
	Repository: https://github.com/facebookresearch/sam-3d-objects
	Website: https://ai.meta.com/sam3d/
	"""
	"""

[Copilot]

The arXiv ID format "2511.16624" appears suspicious. Standard arXiv IDs after April 2007 follow the format YYMM.NNNNN (e.g., 2501.12345 for January 2025). The ID "2511.16624" would represent November 2025, which is in the future. Please verify this arXiv ID is correct, as it may not exist or may be a placeholder. This same incorrect ID appears in multiple places in the documentation.

Suggested change

	"SAM 3D Team (2025). SAM 3D: 3Dfy Anything in Images. https://arxiv.org/abs/2511.16624"
	"SAM 3D Team (2025). SAM 3D: 3Dfy Anything in Images (preprint)."

[Copilot]

Similar to the mask parameter handling issue, the image parameter has a type hint that includes np.ndarray and PIL.Image.Image, but only the str case is explicitly handled (line 204-205). If a numpy array or PIL Image is passed as the image parameter, it will be passed directly to the inference without any processing. While this may be the intended behavior (i.e., the inference expects these types directly), it's asymmetric with how mask handles numpy arrays (converting them to PIL Images). Consider documenting this behavior or adding explicit handling for consistency.

Suggested change

	# Load image if path
	if isinstance(image, str):
	image = load_image(image)
	# Load mask if path
	# Load or normalize image depending on input type
	if isinstance(image, str):
	image = load_image(image)
	elif isinstance(image, np.ndarray):
	# Convert numpy image to PIL Image for consistency with mask handling
	from PIL import Image
	image = Image.fromarray(image)
	# Load or normalize mask depending on input type

[Copilot]

The class docstring lists "config_path" as an attribute, but this is not actually set as an instance attribute in the init method. The config_path is computed in _setup_inference() but never stored. Consider either removing this from the Attributes section of the docstring, or storing it as self.config_path if it's useful for users to access.

Suggested change

config_path: Path to the pipeline configuration.