Skip to content

Add software to filter SfM data and to create landmarks-based masking for depth map computation #1526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 37 commits into
base: develop
Choose a base branch
from

Conversation

almarouk
Copy link
Member

@almarouk almarouk commented Sep 6, 2023

Linked to alicevision/Meshroom#2211

Description

This PR introduces the following two main features:

  • Depth maps computation can be masked using landmarks from SfM data.
  • SfM data can be filtered to efficiently be used for depth maps masking.

The SfM data filtering supports the following features (the parameters are independent):

  • Filtering landmarks over 3 steps:
    • Step 1: Remove landmarks that have an insufficient number of landmarks.
      Parameters: minNbObservationsPerLandmark
    • Step 2: Remove landmarks with dissimilar observations compared to their neighbors in 3D.
      Parameters: nbNeighbors3D, minNbObservationsPerLandmark
    • Step 3: Remove landmarks with worse resolution than their neighbors in 3D. The resolution is based on pixel size.
      Parameters: radiusScale, useFeatureScale
  • Filtering observations to select best observations per landmark and to ensure consistency with neighbors in 3D.
    The observations are scored in function of distance between corresponding camera centers and landmarks positions.
    The scores are iteratively propagated between neighboring landmarks in 3D.
    Parameters: maxNbObservationsPerLandmark, propagationEnabled, nbNeighbors3D, nbIterations, dampingEnabled, dampingFactor
  • Filtering observations (and eventually landmarks) to ensure non-sparsity in 2D projection of landmarks separately in each view.
    Mask radius for projected landmarks per view is estimated and reported in an output text file (outputRadiiFile).
    Parameters: nbNeighbors2D, percentile, maskRadiusThreshold

Future Work

  • fix duplicate landmarks' biased influence.
  • take into consideration the angle of observation when scoring the observations in addition to the distance to camera.
  • include blur estimation.
  • have a more complex relative estimation for neighborhoods (e.g. elliptic neighborhood with orientation, based on landmarks segmentation/classification for surface information...).
  • verify if modifications are required in case of multiple capturing devices.

@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 13 times, most recently from 6851026 to b7ff339 Compare September 12, 2023 15:24
@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 3 times, most recently from 7d09e3a to 21105d0 Compare September 25, 2023 07:53
@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 6 times, most recently from 3002139 to 718224d Compare September 27, 2023 09:40
@almarouk almarouk marked this pull request as ready for review September 27, 2023 09:42
@almarouk almarouk changed the title Dev/mask depth map filter sfm Add software to filter SfM data and to create landmarks-based masking for depth map computation Sep 27, 2023
@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch from 1d586f7 to 27a1b64 Compare March 26, 2024 11:39
- resolve duplicate features issue
- do not normalize propagated neighbors scores
- fix NaN MSE issue
- fix radius estimation caused by quantile issue
- add option to propagate neighbor observations
- use Jaccard index in step 2
at the end of 3D Observations filter
- create a fake observation by projecting the landmark in the view
@github-actions github-actions bot added the stale label Mar 22, 2025
@github-actions github-actions bot closed this Mar 29, 2025
@natowi natowi reopened this Mar 29, 2025
@github-actions github-actions bot removed the stale label Mar 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants