feat: Add SAM 3D Objects support for 3D reconstruction

docs/examples/sam3d.ipynb

@@ -0,0 +1,228 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 3D Object Reconstruction with SAM 3D\n",
+    "\n",
+    "[![image](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/opengeos/segment-geospatial/blob/main/docs/examples/sam3d.ipynb)\n",
+    "\n",
+    "This notebook demonstrates how to use SAM 3D Objects to reconstruct 3D models from segmented objects in images.\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "[SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects) is a foundation model from Meta that converts masked objects in images into 3D models with pose, shape, texture, and layout. It excels in real-world scenarios with occlusion and clutter.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- Linux 64-bit system\n",
+    "- NVIDIA GPU with at least 32GB VRAM\n",
+    "- HuggingFace authentication for checkpoint access\n",
+    "\n",
+    "**Reference:**\n",
+    "SAM 3D Team (2025). SAM 3D: 3Dfy Anything in Images. https://arxiv.org/abs/2511.16624"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installation\n",
+    "\n",
+    "SAM 3D Objects requires a separate installation. Print the instructions:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from samgeo.sam3d import print_install_instructions\n",
+    "\n",
+    "print_install_instructions()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup Environment Variable\n",
+    "\n",
+    "After installing SAM 3D Objects, set the path to the repository:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "# Set the path to your sam-3d-objects installation\n",
+    "# os.environ[\"SAM3D_PATH\"] = \"/path/to/sam-3d-objects\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Usage\n",
+    "\n",
+    "Reconstruct a 3D object from an image and mask:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "# Initialize the reconstructor\n",
+    "# reconstructor = Sam3DReconstructor()\n",
+    "\n",
+    "# Reconstruct a single object\n",
+    "# output = reconstructor.reconstruct(\n",
+    "#     image=\"path/to/image.png\",\n",
+    "#     mask=\"path/to/mask.png\",\n",
+    "#     seed=42,\n",
+    "# )\n",
+    "\n",
+    "# Save the Gaussian splat\n",
+    "# reconstructor.save_ply(output, \"object.ply\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Workflow: Segment with SAM, Reconstruct with SAM 3D\n",
+    "\n",
+    "A typical workflow combines SAM/SAM2/SAM3 for segmentation with SAM 3D for 3D reconstruction:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Step 1: Segment objects using SamGeo\n",
+    "# from samgeo import SamGeo\n",
+    "# \n",
+    "# sam = SamGeo()\n",
+    "# sam.set_image(\"image.tif\")\n",
+    "# sam.generate(output=\"masks.gpkg\")\n",
+    "\n",
+    "# Step 2: Reconstruct 3D models from segmented objects\n",
+    "# from samgeo.sam3d import reconstruct_from_samgeo\n",
+    "# import geopandas as gpd\n",
+    "# \n",
+    "# masks = gpd.read_file(\"masks.gpkg\")\n",
+    "# ply_files = reconstruct_from_samgeo(\n",
+    "#     samgeo_result=masks,\n",
+    "#     image_path=\"image.tif\",\n",
+    "#     output_dir=\"./3d_models\",\n",
+    "#     max_objects=5,  # Limit number of objects\n",
+    "# )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multiple Object Reconstruction\n",
+    "\n",
+    "Reconstruct multiple objects from a single image:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# reconstructor = Sam3DReconstructor()\n",
+    "\n",
+    "# masks = [\"mask1.png\", \"mask2.png\", \"mask3.png\"]\n",
+    "# outputs = reconstructor.reconstruct_multiple(\n",
+    "#     image=\"image.png\",\n",
+    "#     masks=masks,\n",
+    "# )\n",
+    "\n",
+    "# for i, output in enumerate(outputs):\n",
+    "#     reconstructor.save_ply(output, f\"object_{i}.ply\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Output Formats\n",
+    "\n",
+    "SAM 3D Objects outputs Gaussian splats which can be:\n",
+    "- Saved as PLY files for viewing in 3D viewers\n",
+    "- Converted to meshes for use in 3D software\n",
+    "- Visualized in web-based viewers\n",
+    "\n",
+    "For viewing PLY files, you can use:\n",
+    "- [MeshLab](https://www.meshlab.net/)\n",
+    "- [CloudCompare](https://www.cloudcompare.org/)\n",
+    "- [Three.js](https://threejs.org/) web viewer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tips and Best Practices\n",
+    "\n",
+    "1. **GPU Memory**: SAM 3D requires at least 32GB VRAM. Use smaller images or process objects one at a time if you have memory constraints.\n",
+    "\n",
+    "2. **Mask Quality**: Better segmentation masks lead to better 3D reconstructions. Use SAM2 or SAM3 for high-quality masks.\n",
+    "\n",
+    "3. **Object Selection**: SAM 3D works best on:\n",
+    "   - Complete, unoccluded objects\n",
+    "   - Objects with clear boundaries\n",
+    "   - Objects with sufficient texture\n",
+    "\n",
+    "4. **Seed Consistency**: Use the same seed for reproducible results.\n",
+    "\n",
+    "5. **Batch Processing**: For multiple objects, use `reconstruct_multiple()` or `reconstruct_from_samgeo()` for efficiency."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## References\n",
+    "\n",
+    "- [SAM 3D Objects GitHub](https://github.com/facebookresearch/sam-3d-objects)\n",
+    "- [SAM 3D Website](https://ai.meta.com/sam3d/)\n",
+    "- [SAM 3D Paper](https://arxiv.org/abs/2511.16624)\n",
+    "- [SAM 3D Demo](https://www.aidemos.meta.com/segment-anything/editor/convert-image-to-3d)\n",
+    "- [SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) (for human mesh recovery)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "version": "3.11.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/sam3d.md

@@ -0,0 +1,3 @@
+# sam3d module
+
+::: samgeo.sam3d

mkdocs.yml

@@ -83,6 +83,7 @@ nav:
           - examples/sam3_box_prompts.ipynb
           - examples/sam3_tiled_segmentation.ipynb
           - examples/detectree2.ipynb
+          - examples/sam3d.ipynb
     - Workshops:
           - workshops/purdue.ipynb
           - workshops/cn_workshop.ipynb
@@ -98,4 +99,5 @@ nav:
           - hq_sam module: hq_sam.md
           - text_sam module: text_sam.md
           - detectree2 module: detectree2.md
+          - sam3d module: sam3d.md
           #   - fer module: fer.md

samgeo/__init__.py

@@ -21,3 +21,10 @@
     )
 except ImportError:
     pass  # detectree2 not installed
+
+# SAM 3D Objects support for 3D reconstruction
+from .sam3d import (
+    Sam3DReconstructor,
+    print_install_instructions as sam3d_install_instructions,
+    reconstruct_from_samgeo,
+)