about how iPhone streaming to M1 Mac

[yinloonga]

• I'm sorry to bother you. Since I don't have your contact information(Gamil...), I may open an issue here to get your attention. I watched your Youtube video: 3D scanning on iPhone streaming to M1 Mac using vdbfusion and Python
  . To be honest, it's a very good work.
• And it's the first time that I have seen that just use the iPhone as a "Localization" role in SLAM(Simultaneous Localization and Mapping). Amazing, and I also learned that in iphone, by using ARKit can get some important informations：

By comparing the position of the tracked feature differences between video frames, and then comparing with the motion sensor data, a high-precision position and motion model is obtained. (Apple Developer Official website)

• It's apparently from the video that the iPhone's "Localization" feature is excellent, and I'd like to explore more work in this area as well. Traditional high-definition positioning equipment(like Vicon motion capture system) is expensive and complicated to use. Compared with this functions of the iPhone, its light weight and low cost can shorten the development cycle, or even as a reference example that can quickly compare with other SLAM methods.
• However, I rarely use 3D scan on iphone streaming to M1 Mac lively. The implementation process is not well understood. Therefore, I hope that the author, can you provide a code in the Youtube video for learning reference.
  Thank you very much.

[mpottinger]

Hello, sorry I did not see this earlier. I will follow up when I have time. Yes the ARKit tracking is pretty good and more than enough for many uses. I wrote a small lightweight app for iOS which serializes the necessary data to create a point cloud as JSON, along with the depth and color frames in binary, after that the integration with vbdfusion was almost trivial since point cloud and camera position are all it needs.

[yinloonga]

• Thanks very much for your reply. I've downloaded an app from the App Store for 3D reconstruction (iOS device with lidar), and it's amazing. Obviously, you have more experience on ARKit.

• In SLAM, point cloud in 3D space is obtained through sensors (just like lidar, camera, etc). But it is a novel method to

  serialize the necessary data by using iPhone to create a point cloud as JSON, along with the depth and color frames in binary.

• Therefore, I would like to know if you have any plans to open source this part of the code, and roughly when. To be honest, I am currently working on a project, and I hope to use this as a test part(non-commercial use).

• Would appreciate it if you could. I wish you have a happy life.

[mpottinger]

• Thanks very much for your reply. I've downloaded an app from the App Store for 3D reconstruction (iOS device with lidar), and it's amazing. Obviously, you have more experience on ARKit.
• In SLAM, point cloud in 3D space is obtained through sensors (just like lidar, camera, etc). But it is a novel method to

  serialize the necessary data by using iPhone to create a point cloud as JSON, along with the depth and color frames in binary.

• Therefore, I would like to know if you have any plans to open source this part of the code, and roughly when. To be honest, I am currently working on a project, and I hope to use this as a test part(non-commercial use).
• Would appreciate it if you could. I wish you have a happy life.

I am carefully considering whether or not I will release the code or what parts I may release. I suppose the streaming code is not too complex, though yes it does require some experience with the AR SDK's like ARKit, ARCore, etc. The project I'm working on is for my workplace and may end up being a commercial project, so I have to consider what I'll be releasing. Definitely I am open to giving any assistance and I will let you know.

[yinloonga]

OK, thanks.
I will keep following up. And if there is any latest progress, please notify in time. This is my Gmail: yinlonga@gmail.com